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ABSTRACT 

Proceedings of a workshop on program review in 
postsecondary education and a report of a program review study are 
presented. The workshop was designed to acquaint participants with 
the changing academic planning environment, to pi^ovide a better 
unisrstaadino of the purposes of program review, and to excHings 
ideas for improving procedures. Its fosiis wa)5 on t)rocedures in the 13 
western states; the emphasis was on ^he r elat ioTfshi^) of state agansy 
practices to institutional program review^ The/ presentations cover 
CJte*-xaTrge-~froifi theoretical to actual practices, from the context in 
which progtam review will continue to take pllace^tp the use of 
quantitative measures to evaluate quality. The report is the result 
of a survey of institutions and state higher/ education agencies in 
the 13 western states. It examines the issuej^ involved in program 
review, describes the approaches taksn to it,\ and clarifies the 
differences in program review activi*:ies al t 
institutional levels. State profiles are give 
California, Colorado, Hawaii, Idaho, Montana, 
Oregon, Utah, Washington,' and Wyoming, Tables 
responses by level; involvement in development 
purpose for review: criteria for review of new 
date prasent policies were initiated: criteria 
progwups for review; criteria for review of existing programs; 
relation of program review tr> other activities, (SW) 
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Fprewor'd 

Program review is an important activity in the management and planning 
of higher education today. It is crucial, in the face of changes in student 
demand and increasing fiscal constraints/for institutions and state agencies 
to assess both existing and proposed programs with an eye toward the most 
effective use of resources. Program review is a tooV which, used wisely, can 
effectively respond to the need to maintain ,qual ity even in the face of the 
current pressures on higher education. 

This workshop was designed to acquaint participan'ts with the changing 
academic planning environment, to provide a better understanding of the 
purposes of program review, and to exchange ideas for improving procedures. 
Its focus was on procedures in the thirteen western states; the emphasis 
was on the^relationship of state agency practices to institutional program 
review. A variety of perspectives was presented among tlie speakers at the 
conference. Together, their presentations cover the range from the theoretical 
(Edward Kelly) tc actual practices (all the presentations in section ' three) , 
from the context in which program review wi"l 1 continue to take place (section 1) 
/to the use of quantitative measures to evaluate quality (William Chance). 

^Workshop proceedings constitute Part I of this publication. Readers who are 
interested^.m a summary of the proceedinos should refer to the first section 

Vin_thfi proceedings, "Workshop Overview" by Jphn Fblger. 

Part II of the publication is a report of a program review study conducted 
by Lilla Engdahl of the WICHL staff and Robert Barak, Director of Research and 
Information, Iowa State Board of Regents. This report is the result of a sur- 
vey of institutions and state higher education agencies in the thirteen western 
states. It examines the issues involved in program review, describes the 



approaches taken to it, and clarifies the differences in program review activi- 
ties n the state and institutional levels. A comprehensive discussion ef 
actual practices used in the review of higher education programs in the West, 
this report should be of interest to decision makers at both state agencies 
and higher eii'<:ation institutions. 

* 

We would like .to thank the California Postsecondary Education Commission 
for hosting the workshop and Barbara Krauth for coordinating workshop arrange-, 
ments and editing this publication. The work of Cheryl Pedersen and Dorothy 
Read of the WICHE staff in typing the manuscript and preparing meeting materials 
is also appreciated. 
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WORKSHOP OVERVIEW 
John Folger 



The conference made a creative contribution to our understanding of 
the purposes of program review, and to the future prospects for this type 
of activity. I-t began with a review of the environmental conditions within 
which program review will be made, and which will shape the expectations 
for this activity. It was followed by discussion of the purposes of program 
review, and this led into discussion of the future of program review 
activities. Thi^ summary organizes the conference discussion under these • 
three major headings. 

The Environment ' • 

The first session explored the environment for program review. There 
was general agreement that: enrollments will decline nationally and in 
most states by about 10-15 percent; legislators will be unwilling to provide 
any real dollar increases for higher education in most states and in some 
states, appropriations may not keep up With inflation; higher education 
had better get busy and do its own prografn review before somebody else does 
it for them. 

The opening session focused on trends in enrollment, but the important 
problem is going to be fluctua.tions around those trends. The demographic 
trend indicated an average loss of 2 percent a year in enrollment between 
1980 and 1985, and another 1 percent a year between 985 and 1990; but, 
those average trends do not indicate that there will be wide , variations 
from year to year about these, trends, and the variations may create more 
problems than the trends themselves. 

An equally likely trend is that enrollments will decline only an average 
of 1 percent a year between 1980 and 1985 and not at all during the next five- 
year period. That prediction is based on the following points: The popula- 
tion between the ages of 25 and 40, with some college experience, will increase 
about 33 percent by 1990; this represents a sizeable increase in the potential 
-narket. While this group makes up about one-fourth of the enrollment now, 
it might comprise one-third by 1990. The second point is that colleges ha^e' 
a great deal at stake in increasing participation rates and it is likely 
that they will do so, either by attracting new clientele or by serving 
existing clientele in ways that enhance their budgetary situation. 

Between 1974 <and 1975 enrollment jumped 10 percent as a result of more ^ 
part-time attendance during a recession. Between 1975 and 1976 it leveled 
off. Annual fluctuations of 10 percent up or down around the trend are 
quite possible; this can lead to crisis management of finances .ind possi- 
bly of programs. 

Long-term real growth in income averaged about 3-4 percent a year in 
the 1950's dnd 1960's. It has averaged about 1-2 percent a year in the 1970'sj 
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and can easily fluctuate around that trend by 3^5 percent a year. This 
causes even larger swings in tax collections and in the discretionary 
state revenues that higher education can compete for. A reasonable pre- 
diction is that we are likely to have at least two recessions in the. next 
decade, and that both fluctuating revenues and fluctuating enrollments 
are likely to disrupt orderly planning and program review, . 

The projections vary substantially from state to state. About 5 or 
6 states will have increases in the college age group between 1975 and 
1990, while a few states will. have decreases of 2b percent or more. The 
average decrease of 15 percent contains wide state to state variations. 

■ _ • ©, 

Within-state variations in enrollment will be even larger than varia- 
tions between states. It will not be uncommon. for states to have some 
institutions that lose 40-50 percent of their enrollment, while other in- 
stitutions in the same \state grow. Between 1970 and 1975, 10 percent of 
the public institutionsMost over 10 percent of their enrollment, averaging 
about a 22 percent loss, but the ^/erage public enrollment' increased about 
30 percent during those same five' years. ^ 

To summarize the session: planning, program review, and budgeting will 
have to be done under conditions of great variability. The pressure'^to avoi^ 
complex evaluation procedures in /favor of simple crisis management ^tivitie^s 
such as across the board adjustnients will be great. The question for higher 
education in some states will be: with legislatively enacted revenue and 
expenditure controls on the budget, will anybody pay attention to program 
review as a management or fiscal control process? 

The Challenge 

Patrick Callan emphasized the importance of states and institutions 
having a plan, or plans, for dealing with the environmental conditions of 
the 80's, rather than reacting from crisis to crisis. A conservative plan 
will be more use.ful than trying to operate under conditions of great 
uncertainty. 

Purposes of Program Review 

This conference ma(de a considerable contribution to clarifying the 
• purposes that institutions, state agencies, and state legislatures and 

officials may-have for program review. Let me state some of the "purposes 
v'J^' that emerged. 

For institutions, the traditional purpose has beep: 

• A self-assessment for improvement initiated within the institution 
to provide some judgment about the worth and needs of programs. 
Accreditation program review has also been carried out largely in 
the spirit of self-impi ovement. On-ly in the marginal institutions 
does accreditation have a major threat of sanctions. 
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A. new purpose is : 

• To reallocate resources within the institution, to enable the 
institution to do more with the available resources. The collegia! 
nature of institutional governance makes this new purpose difficult 
to achieve. We have a tradition of widespread faculty participation 
in program review activities, but if the purpose is rejource reallo- 
cation, faculty participation is more likely to thwart or impede the 
process than to make it acceptable. Traditional notions that wide- 
spread participation is important to implementation need to/be re-, 
assessed if the primary- purpose is resource reallocation . 

Callan stressed the- importafnce of removing disincentives and adding 
incentives to institutional administrators to' undertake the difficult task, 
of program review and reallocation. Dick Jonsen -added the important idea 
that program review 'as a resource reallocation process might make more 
headway if it did not deaTwith traditional disciplinary definitions of 
"program." The' discussion suggested that external groups, such as legisla- 
tors or state budget officers, are not concerned about the disciplinary 
mix within an institution, but instead are concerned with effective manage- 
ment and service to students. However, external groups are cpncerned with • 
the profe'isional programs that^are offered, ,so that aspect of program review 
is like4-y -to command considerable external Jnterest. \ 

This point needs further examinatioiv The faculty are, first and fore- 
most, disciplinary specialists, and if program. review could focus more on 
the adjustment of curricula to a changing student clientele or on an evalu- 
ation of the impact or education on students, it might ma'ke more headway 
within the institution- than an examination of programs simply as disciplines 
and profession:! specialties. 

It was also pointed out that internal " institutional program review has' 
to be linked to tl^^e budget process and to personnel decisions if it is to 
be effective as ei the r a reallocation process or a renewal and improvement 
process. . ^ 

The conference discussions also examined the purposes of .program review 
for the state agency. 

The historical purpose was to provide for. orderly growth by controlling 
institutional expansion through approval of new programs. That purpose can 
be extended to the present, where approval of new programs can be used to 
control wasteful or excessive competition among institutions for a shrinking 
or static pool of students, 

A second state purpose is simiUr to the institutional purpose; 
to provide for quality control, to prevent substandard programs, and to 
encourage good ones. Many institutions resist the state role in qualitative 
judgment, however, and in some states their purpose in program review is 
expressed as "seeing that each institution has an effective process of in- 
ternal review." Accreditation is the traditional vehicle for quality control 
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through self- regulation. Unfortunately^ the_credibility of accreditation 
at the state level is low; most institutions trying to persuade the state 
that they have an "effective process of internal revi^ew" are going to have^ 
to add to the usual accreditation review process. 

A third state purpose is to assure that, statewide, the mix of programs- 
represents the most effective use of available resources. Are the programs 
properly related to stuoent demand and society's demand for graduates, and 
are they a reasonably efficient size? Has the institution's quest for stu- 
dents led /it to fragment its efforts in too many areas and blur its role 
and mission? The importance of clear role and missions-statements as a' basis 
for state program review was stressed. With clear and concise role and mis^ 
sion statemerits, program review is simplified. The California State Colleges, 
annual program plan, the Washington program registry, and other similar means/ 
of maintaining a specific program statement for each institution serve the 
purpose of state program review. Role arid mission statements should be the .' 
basis fpr determining, the extent to which the programs offered in the state ' 
are consonant with neejls of the citizens/students and employers. 

A fourth state purpose for program review' is to assure state officials 
that resources are being used efficiently and that institutions are dealing 
with problems of overlap and duplication in programs. The basic state in- 
terest is in efficient 'management, of which program review is only a part. 

Some of the most difficult issues in state program review arise out of 
the merger of these different purposes. In practice, a state's program re- 
view efforts may represent a blend of these various purposes, with an empha- 
sis on a credible management of program resources becoming the most important 
in the next decade". 

The issue of the state agency as institutional advocate vs. the state 
c-gency as critic and evaluator was discussed without being resolved. Coordi- 
nating boards have been able to establish themselves as more credible than 
governing boards i^n the evaluator role, primarily becauseXthey do net play 
the advocate role' as aggressively as the governing boards. \ 

The Future Directions for Program Review 

Pat Callan emphasized the need for the state, through the state agency, 
to establish some stable planning assumptions so that a program review proc- 
ess has some chance of success. Sudden political demands for organizing all 
programs in priority order and eliminating those that are lowest in priority 
will have a "chilling effect" on an orderly process of planning, program 
review, and resource reallocation. 

The state agency role in program review during the next decade is more 
likely to emphasize regulation and resource reallocation than facilitating 
institutional "self-renewal and improven>ent." But if the state agency can 
provide incentives by dollars for limited strengthening of selected areas 
••.uch as the Florida Center of Excellence concept, it may encourage the self- 
renewal process within the institutions. 



9 

If 



Finally, the state interest of legislators and officials Is likely to ^ 
center on the budgetary impli rations, ratheV- tnan directly on program review. 
It was pointed out that general l^legislators and state officials are not 
interested in proyram issues, except w.'iere these invo've high visibility 
programs, such as medical schools, termination of dn engineering school, or" 
institutional closure. Where legislatures have become directly involved in 
program review, as in Colorado and ilebraska, results have not been politically 
rewarding and aren't likely to be in the future. State agencies will* take 
tho heat in most stat^for program reviews that art palitically unpopular. 

The discussions brought'' oat the i importance i^: for the state* agency as 
•weil as the institution, of the linkage of . pliaM'rig, program review, and 
buqgeting. o 

Finally, the conference participants identified some program review 
issues that need further resolution: 



1. How best to link program ard budget? 

* Does it imply a program budget approach? 



What budget incentives to self- renewal and program 
improvement can be provided? \ ■ 



• Will adoption of "priority budgeting" or "zero based 
budgeting" ^limit the interest in program review by ' 
substituting a quick budgetary eva-luation of programs 
for a more carefully design d process?" 

2. What are the best ways of relating a program 'feview process to 
planning and to the role and missfon delineation? 

3. Whdt are the best relationships betv/een. state ag.ency program 
re.view and institutional and state purposes>--both formative 
evaluation and suninative evaluation objectives? 

$ Can states develop and enforce criteria for institutional 
[)rograi'i review so that institutional program review serves- 
state purposes adequately? ' " 

4. Ho.v can^incentives be provided to institutions to engage in 
effective renewal and improvemet^t activities? Which current 

program review procedures foster renewal and improvement? (J 

J. What is the best way for states to demonstrate i to the public 
that thfy and the institutions in the -state have an effective 

iiiiinageii„'nt of program resources? 



Thi" "(.on^crence served a useful purpose in raising ihese issues and 
Mii,(|P'. lifK) approaches to t'lrm. The answers to these questions will neces- 
■.jni/ viry from state to, tate, because of the difference in traditions. 
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existing structures, and the natur^ of the enrollment and f'scal problems 
in' the state. But all states can benefit from an orderly planni'ng process, 
clarification of purposes for program review, and definition of the roles 
of institutions, state higher education agencies, legislators, and governors 
in the process . 
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A discussion of the demographio and fisoal ^environment 
in which program review ie currently taking place. 
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DEKOGRAPrilC CHANGES 
Larry Suter 

Demographic changes in the' United States have always been closely tied to 
postsecondary enrollments, and I bplieve this relationship will be even 
stronger in the future. I have been asked to. address the question of what 
is likely to happen to. postsecondary enrollments in the next ten years, 
and will try to state my opinion as succinctly as possible: I think that 
enrollment in colleges will decline by about lb percent by 1990. It will 
decline by approximately 10 percent by 1985 and then continue downward. 

Past enrollment increases were due to three main 'social and demographic 
forces: 



• Population growth between 1947-1961. The baby boom began in October 
of 1947; following tHat, the number of. births fluctuated until 1961. 
The high birth years were 1959-1961, after which the absolute number 
of children born declined and has remained low. 

i Increase in high school graduate rates. In the 1950's, only about 
60 percent of the eligible age group graduated from high school. 
By 1970, that rate had increased to 75 percent; by 1975, it was 
85 percent, where it levelled off. In the West the graduation rate 
is at about 90 percent, whereas the rate in some states in the South 
is only 68-75 percent. 

• Higher proportions of men and women high school graduates attending 
college. During the Viet Nam war some. men--about 10 percent of 
the age group--went to college because of draft deferments. Women, 
also, began to attend college at increasing rates. In. the last few _ 
years, women aged 18-19 have been attending college at nearly the 

4^ same rate as men. 

• More older persons attending college. Although two-thirds of 
undergraduates are still 18-21 years old, the proportion of older /' 
persons attending has increased, . A number of predictions suggest / 
that the increase in enrollment rates among older women, in / 
particAJlar, is likely to offset other factors to counteract decliries. 
I dc^ot believe that it will. \j 

These same factors will dominate future enrollment patterns, but with 
the following differences: . 

• There will be no further increase in high school graduation rates, 
for most states. The greatest potential change is in the South; 
the rate is unlikely to go above the present 90 percent in the West. 
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• There will be a levelling of the ratio of high school to college 
« rates. There is no good explanation for this other than a need 

for immediate employment and vthe changes in the perceived benefits 

of college. ' 



The population as a whole will decline (population in thousands):. 



Ages 


1980 


1985 


% of 
1980 


1990 


% of 
1980 


15-24 


41,527 


38,517 


92.8 


34,730 


83.6 


18-21 . 


17,117 


15,442 


90.2 


14,507 


84.8 


22-24 


12,346 


12,411 


100.1 


10,642 


86.2 


25-29 


18,930 


20,581 


108. Z 


20,169 


106.5 


30-34 


17,242 


19,278 


^ 111.8 


20,917 


121.3 



Graduate enrollments should of course decline, as well, especially - 
to the extent that graduate schools have served to prepare students 
for academic careers. Enrollments should decline significantly in 
the 1980's; however, b.irth rates could pick up in the 1980's for 
a brief period, creating more potential coVlege students in the 
years 2000-2010. 
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FISCAL CONSTRAINTS 
Harold Geiogue 



Proposition 13 is a year old in California, but it has become a national 
phenomenon in the course of that year. We have had some lessons drawn for* 
, us which might be useful to those of you" in other states where higher educa- 
tion is likely to be affected by constraints' on government expenditures. 

Proposition 13 is, of course, not an expenditure' limitation, but merely"^ 
a limitation ion one source of revenue, the property tax. Propositi oi^ 13 
doesn't say anything about whether the government can grow 2p or 30 or 
40 percent; it just says that they cam't fund it more than X from the prop- 
erty tax. Because of this, its main impact was on the coimunity colleges. 
Coirmunity colleges in California have traditionally been 60 or 70 percent 
funded from the local property tax, with the remainder coming from the state 
and a very smaN percentage from the federal government. So it was clear 
that the conmunity colleges would have to adjust their planning to take 
account of fiscal constraints. ^ 

More interesting^ however, is the fact that Proposition 13 has had a 
direct impact on the University of California and the California State 
University and Colleges, which have had no property tax support. In fact, 
the most bewildered people after Proposition 13 were these segments of 
postsecondary education. What developed, obviously, was a political en- 
vironment involving the fiscal condition in general and' not just the ' 
property tax or any other form of tax. The general voter climate of opin- 
ion was (and is) against government spending in'all forms. 

The impact of that attitude was felt. at all levels last year, particu- 
larly in postsecondary education. Since we have a July 1 deadline for the 
budget, we went through all the budgets in about 20 days and reduced them 
in a rather severe fashion. Major reductions were made in operjating expenses, 
and all the new programs on line were taken out. Salary increase items, of 
course, also fell out of the budget. 

' » 

' Following that, the mood continued into November, when a significantly 
new group of legislators was elected--in the wake of Proposition 13 and a 
new political environment. Several members who had expected to return to 
the leqislature this year did not because Mr. Jarvis sent out a computer 
letter {which arrived the day before the election) saying, in essence, 
"Thanks for voting for Proposition 13. The person I think most carries 
my message is Joe or Jane Blow." Several incumbents were "bushwhacked" 
in that manner. This phenomenon may spread to your states as well-, it is 
certainly a reamy in California. 

\^ The general! impulse to jump on the bandwagon has also caused two new 
iniNtiatives, onj arKexpendi ture control initiative, the other another Jarvis 
prop\sdl--dn i ntome "ta^ limitation. The- governor, too, has certainly heard 
the in^sagu anU-ki^s/been going around the country talking about expenditure 
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control and deficit spend* nig. The irony here is that California itself is 
in a deficit spending pattern. While we are proselytizing to the nation, 
we are currently sinners ourselves. Another reality in our poli tical en- 
vironment is reapportionment. . Generally over the course of the legislative 
sessions, the concern is with budget and taxes, but every ten years, it's 
reapportionment. Given ^the mood of the public, I believe that the next major 
election will lean heavily towards fiscal issues. 

One final reality: |In terms of postsecondary education's share within 
this constrained fiscal e|nvi ronment, we need to keep in mind that spending' 
^at the state level has two major functions. One is to provide local ' 
* assistance— for cities, counties, special districts, even school districts. 
The second is what we call the major state operations. We have a total 
budget of 15-16 billion dollars, but only a third of that is the control-\ 
Table state operations; the rest goes out under local assistance formulas. \ 
When we look for economies in the budget, postsecondary education has a \ 
major share of the state operations part of our budget. Postsecondary ^ 
education does stand out, therefore, as an available target, and it Will 
be looked to more and more as a place to find economies. We feel that 
major trade-offs will be questions of access and questions of quality, i 
Institutions will either increase productivity and workload or in some way 
take advantage of the decline in enrollments as ways to control cost. 

In the 14 years of my experience in Sacramento, there has been a great 
deal of rhetoric about economy and fiscal constra^ints. Nevertheless I be- 
lieve tha.t the lesson of Proposition 13 is that the reality is now upon us; 
economy and fiscal constraints are no longer only the rhetoric of politicians. 
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PROJECTIONS OF HIGH SCHOOL GRADUATES 



William McConnell 



I have recently completed a report which projects the annual number 
of graduates of the public high schools in each of the thirteen western 
..states through 1995; The results of that study indicate that the widely 
publicized declines in the college age population which are projected to ^ 
occur in the near future are likely to vary in degree frop state to state. 

/ Declines in western states are not expected to be as severe as in 
the nation as a whole, although they will follow the -same general pattern: 

• a peak between 1975 and 1982 ^ 



% a significant decrease from that peak, levelling \ff to a low 
point between 1984-86 > 

% a brief upturn in 1986-88 . 

• a decrease to another low point in 1990-91 

• an increase to 1995, the last year of the projection. 

The implications of this situation for higher education planinng and 
the review of existing programs are quite clear. Projections of high 
school graduates in each state are part of an increasingly important effort 
to plan for shifting enrollment patterns. Obviously, it is particularly 
important during a period of such varied roller-coaster falls and rises 
in the number of high school graduates to carefully monitor developments 
state by state. In Wyoming, for example, where the projection shows 
a decrease only between 1982 and 1984 fol lowed by a steady increase to 
1995, ^ but the picture in California is quite different. 

B^efore I show you the individual state projections of 'ligh school 
graduates over the next fifteen years, I'd like to say a little about th^ 
iriethod \^sed in deriving them. The, projection technique used was the 
grade-progression method. Using historical data (on births, grade by 
grade eri^rol Iments , and high school graduates), the ratio of enrollment 
in first\grade to number of resident live births six years earlier was 



tnent one year-to first-grade enrollment the D'evious year was calculsted 
for each a\ge group. Similarly, a column of 'ios was calculated lOr ' 
each move from grade to gr'ade through twelfth ^rade. 

The results are as follows: 





The ratio of second-grade Enroll- 
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Pattern of Annual RcildenI Live Births 19591977* 
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INSTITUTIONAL RESPONSES 



T. K. Olson 



As I have talked to my colleagues in other states, I have discovered 
that Oregon— while it vaunts itself as coming out with bold and imaginative 
schemes--appears to have walked some of the same trades as other bold and 
imaginative states. In the state of Oregon we are going through an effort 
to redefine the role and mission of institutior>s in order to bring them 
into line with the new realities of enrollment decline. No doubt you are 
doing the same. It's amazing, though, the way you can scratch around and 
find an old piece of parchment -somewhere that says that even though you 
didn't offer X or Y kind of course before now, asking for it is not a 
180-degree turn since it has been an historic role for that institution. 
It has never been exercised before, but now, with the new felt realities 
and the grass roots demand for this unique program, it is appropriate. 
I've heard that appeal from universities and colleges anc community colleges. 

The shifts we have noted in Oregon have been the same ones, I am sure, 
that you have seen: an increased interest in graduate students as opposed 
to undergraduates, partly because, in a, public institution, graduate fund- 
ing provides more money than undergraduate. Every time a graduate student 
replaces an undergraduate, you've added substantially more dollars. If you 
can manage to create a graduate program that's less expensive than some 
other graduate programs, then you've made a net profit. If you make the 
mistake of adding a graduate program, that is so expensive that the state 
reimbursement does not cover it, however, then you're in the hole. So 
you have to be imaginative about the graduate programs you propose and 
imaginative in the ways you approach the instruction of students in high 
cost graduate programs. 



There has clearly been a significant increase in the number of part- 
time students, and that affects program reviews in a way I will mention fn 
a moment. And certainly the age cohort is shifting upward in all our 
institutions. Even in the independent colleges in Oregon--which traditionally 
have had almost only students in the 18 to 20 year old age group— the age 
level is increasing. This is partly due to returns from Viet Nam and 
partly .to the fact that student financial aid works to the advantage of 
independent colleges as opposed to public. It is also true that some of 
the independent colleges in Oregon have been in the forefront of offering 
external degree programs specifically aimed at unique clienteles, including 
older students. 

Continuing education is" having an impact on enrollment, of course-- 
in the historic credentiall ing of teachers with a need to be brought up 
to date in new research and techniques, in the drive of salary scales 
predicated- on X number of hours taken at the post-baccalaureate level, 
in new occupations being created, some of which require a continuing 
education, and in. occupations, such as pharmacists or CPAs, which under 



or 
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Oregon law are required .to have X number of credit hours in order tQ 
continue with certification or a license to practice. These people are 
all ending up at the. doorsteps of institutions, and it is difficult to ' 
see institutions becoming reluctant to see more of thefn arrive. 

There is a, shift away from teacher education at the undergraduate level 
and an increase, at the graduate level. There has been a decrease in lower 
division collegiate enrollments at the community college level and an in- 
crease in vocational enrollments, and perversely, an increase in the sort 
ofL^blah" category—the catpgpry v.-hich i? neither lower division collegiate 
nor occupational, but which is called "other reimbursable" in Oregon. An 
"other reimbursable" includes everything from adult basic education* GEO, 
and English as a Second Language, on the one hand, to famous courses like 
cake decorating, on the other. Cake decorating may be a rec national course, 
in which case it is not reimbursable (cake decoratintj for Clyde, who wants- 
to suddenly try h^'s hand at a flourish of making cakes for his wife), but 
it may also be a course to train people to become cake decorators in Ajax 
Bakery. It can fall in either category. 

So those are the trends we have seen--and everybody else has seen. 
Let me mention some ':hings now that we are looking at in Oregon wh^ch 
perhaps many of you are also I'ooking at. One is credit for^prior learning. 
We have talked about the impact of what is happening in teriis of enrollment, 
and this certainty affects how we examine programs. To the exten^that 
institutions grant credit for prior learning, it reduces the ^amount of 
time students spend in school, and therefore it automatically has the 
"impact of "reducing enrollment. In some cases, of course, it provides a 
kind of inducement for attendance. Perhaps you wouldn't have gone to 
school because it takes too long to get a degree, but here at S.igapoo U, 
we will -give you credit for that marvelous life experience you had working 
in a foundry or helping with the Hottentots, or whatever. Then you might 
enroll for a degree where you would not have done otherwise; that can helff 
enrollment. We have great difficulty in trying to sort out whether or 
not to approve a particular program which has a significant element of 
credit for prior learning built into it. 

Another category I am interested in is technological changes. Radio 
was supposed to transform education marvelously and we were no longer 
going to have campuses. It didn't. Then we Jia t'^ievision, which was 
going to transform education and dramatically r \ enrollments; it didn't. 
Then we had the computer and the computer wa:> ^ to dramatically affect 
enrollments and bring education to the home; ana it didn't. I'm wondering 
if maybe we couldn't begin to use some of the^e things effectively, includ- 
■ ing that marvelous instrument called the telephone, and the video tape « 
recorder. Let us postulate for a moment that with modularization of courses 
taking place at the institutional level; we are breaking down more and more 
the components of a course into tiny little parts. Each student comes in 
and takes a pre-test to find out wh. parts to take and then takes only 
the module he doesn't know. Then he passes that course and marches on to 
. the next one. To the' extent that that's occurring somewhere other than 
in medical schools or community colleges, it is happening in business. 
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It is possible that big business, Westinghouse and General Electric and . 
IBM and Xerox, who have no fears of training their own employees through 
the use of technology, may in essence bypass the universities and colleges. 
They should be able to provide academic credential 1 ing for a variety of 
courses and eventually, degrees. If we are postulating about the 1990's, 
why not talk about that? Why not talk about a bypass of the historic 
gatekeepers of. education by the people who have the skills and the tech- 
nology to do it. Can't you imagine the same people who turned out "the 
Beverly Hillbillies" turning but a new, course in music theory? It does 
boggle the mind. ' 

Another area we are looking at is program specialization. Increasingly, • 
our state is looking at the problem of whether we can afford a certain range 
of programs ,?nd whether we should encourage specialized institutions to 
establish themselves for specialized purposes. We have one experi'.ient going 
on in the state of Oregon right now that we will be watching very closely. 
It is a new graduate school for psychology, the only one of its kind. Not 
a dime of state money is going into it, just whatever federal monies 
they are able to gain in terms of research grants. You have often heard 
disgruntled academics talking about doing that . . . "By cracky, I'm so 
sick and tired of this blankety-blank I'm going through, I'm going to set 
of the Snort School of Economic Studies, and I'll have such a good school 
they will come and beat down my doors and I'll enroll them and bypass this 
whole thing." To the extent that chey can succeed in getting accreditation 
or cerMf ication, such centers could affe(ft us not only in terms of. funding 
but in terms of the^»vhole approach to certain kinds of education, graduate 
as well as undergraduate.. Tha.t olso leads into the question of reciprocity. 
Oregon and Washington are engaged right now in negotiations about reciprocity. 
We will be talking not only about tuition reciprocity, which eliminates the 
state border, but, I suspect, about programmatic reciprocity. That is, 
on a bilateral basis, vve will be using the WICHE approach of looking at 
certain programs that we will not promote in Oregon- and will want to take 
advantage of in Washington- -and vice verso, at all levels, not only U 
graduate institutions but at specialized undergraduate and comnunit> 
colleges as wel 1 . 

Increasingly, we as a state agency are looking at shared programs as 
ci first criterion in p^^ogram review. The burden of proof is on institutions 
to Justify why they want to do it alone. Secondly, increasingly, we are 
asking the question of why we, as opposed to other states, 'should offer a 
par;:icu}^^r program. And thirdly, why should we do it all? Either let the 
private sector, or perhaps fir.d ways to approach it through technology to 
avoid our having to i^nshrine it in an academic institution. And then, 
lastly. .1 touqh look at credential! ing, and the reasons for enrollment in 
dll forms of prQqrdiris--in order to see if business, industry, or government 
■should foot the bill. 
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The Challenge to Program Reviewers 



Reoon -iling a responsiveness to fisaal and demogra^' 
vela ions with a aonaern for quality, 

Patrick Callan 

Director ^ ' 
California Postsecondary Education Commission 



^ THE CHALLENGE TO PROGRAM REVIEWERS 

^ • Patrick Calkn 



The subject of program review, particularly the assessment of exist- 
ing academic programs, is rapidly emerging as one of the central issues 
that will confront, institutions of higher education, multi -campus systems 
and states in the 1980s and 1990s. My remarks will attemi^t to place this 
issue in/the context of higher education planning and management, and to 
offer a perspective which may serve as a starting point for your discus- 
sions during this conference. 

We know that the environment' for higher education in the eighfiA 
and beyond will differ significantly from that of the last two decades. 
The impact of lower birth rates and changes in the characteristics of 
the population with respect to age and racial distribution will be felt 
by our colleges and universi ties' beginning around 1983. We must plan to 
Qope with these changes whi'e, at the same time, dealing with the public's 
disillusionment with government and the growing movement to restrict 
revenues and expenditures. This movement may have run its course by the 
mid-eighties, but it may leave in its wake a series of constitutional and 
statutory constraints on government spending, leaving planners to face 
. simultaneously the twin problems of enrollment decreases and fiscal 
constraints. 

Ironically, it is fortunate for everyone but the planners that the 
era of most rapid expansion of higher education is over in almost all of 
our states. Our state systems of higher education are reasonably well- 
developed-"perhaps overdeveloped in some instances. If the emerging 
fiscal constraints had been imposed in the period of rapid expansion, 
the problems would have been insurmountable. Unfortunately for higher 
education, because we will generally not experience significant growth, 
welare likely to be at a competitive disadvantage in. bidding for avail- 
able resources in the public sector, as the demands of other "claimants-- 
social services., energy, environment, health, etc.— continue to grow. 

t 

Perhaps the most significant long-.term consequence of these in- 
creasing limitations on .educational spending is that many states will 
not have^ the Rapacity to cushion the impact of enrollment declines. 
In a more optimistic time, we might have viewed a period of enrollment 
stability and decline as ,an opportunity to argue for new resources to 
enrich programs or to improve staffing ratios. While I am sure that 
such efforts will be made, I doubt that the resources will be available. 
Any state, system, or institutional strategy which assumes that new re- 
sources will be available to tide colleges and universities over the 
problems of the next ten to fifteen years is unrealistic. 

Th^s changed environment is not unique to a few s-tates; it is not 
something that is going, to happen to "tho oth'^-r- guy. " The differences 




ERIC 



28 



from state to state will be differences of degree and magnitude. While 
there are imponderables, all the indicators we have point in one direction: 
five and ten years from now, in most of our states, the systems of higher 
education will be smaller than they are today— in some cases, considerably 
smaller. None of the indicators mitigate this future. I suspect we have 
all indulged in hopes and speculations of windfall inmigrations coupled 
with possible changes in manpower demand Which might increase college par- 
ticipation rates. And each of us nurses the fantasy that the inevi tattle 
will befall the rest of the world, but that our state (or institution) 
will be miraculously spared. 

I a;n not suggesting that we know precisely what will happen. Even- 
in the enrollment area, where we have such indicators as the size of 
various age cohorts, participation rat6s, and high school graduation 
rates, there are still a number of other factors we simply cannot pre- 
dict, including the numbers of part-time versus full-time students, the 
credit loads that students will carry, and so forth. Admi-ttedly, our 
capacity to forecast is limited, but I would remind you of Peter Drucker's 
point that the reason that we need to plan, is precisely because we cannot 
forecast accurately. 

In the absence of certainty, institutions, systems,- and states which 
ignore the obvious, if not -compelling, indicators do so at their peril, 
even though that is, for the most part. Just what most of us haye done. 
We cannot be faulted for. the failure to anticipate the acute fiscal pr6b- 
lems created by the widespread "taxpayers revplt," but the enrollment and 
demographic "surprises" were packaged eighteen years ago. We have to 
ask ourseTves how we in higher, education could have watched elementary 
schools close and enrollments in our public schools shrink without infer- 
ring the eventual impact on our colleges and .universities. 

I do not condemn, out of hand, the speculations *and the hopes that 
enrollments might go . up instead of down, or that public funds might in- 
crease dramatically. There is nothing. wrong with such diversions unless 
they distract us from the facts and our responsibility. If we haVe a 
plan for a line that goes up, that's good clean fun; but if we "don't 
also have a plan for a line that goes down-, that's irresponsible. 

I would suggest that this is a time for "bearish" planning based on 
conservative assumptions, and that at the heart of this planning lies the 
issue of program review. Program review is the cornerstone of the plan- 
ning structure we will need to weather the financial and enrollment storms, 
the 1980s and 1990s, and must provide the central focus of our planning 
and manageiTient. It is important to note that this is a departure from 
what wp have faced in the past— even for those who have already experienced 
temporary enrollment downturns and fiscal pressures— because we are now 
confronted with sustained fiscal and enrollment pressure. It is essential 
that we distiiiguish the ways we have dealt with short-tenn problems from 
the strategies needed to respond to long-term problems. In the short term, 
we are able to use expedient measures, to take emergency steps, to make 
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across-the-board cuts, and to avoid qualitative judgments,- In the long 
term, a more sophisticated strategy is required if we are to preserve 
and enhance quality and access. If we allow our administrators, faculties, 
and governing boards to view this situation as one which will correct it- 
self next year or the year after, if we encourage wishful thinking and 
deferred realism, we will be "setting up" our institutions for a series 
of unpleasant surprises. 

Our ability to maintain contwfol of our own destiny depends in. large 
part on whether we are forthright about the problems and issues and whether 
we move to implement more rigorous program reviev% particularly those of 
existing programs, because politics, like nature, abhors a vacuum. We 
cannot afford the perception on the part of the public officials that 
we are . failing to deal with our problems at the institutional, system, or 
state agency leVel. That perception in itself is enough to trigger in- 
creased governmental intervention in higher education. There are two sets 
of issues here. One is internal--what we do to maintain stability, qual- 
ity and access during a difficult period. The other is how we deal with 
the external environment. I suggest that realistic plannirig is^ the appro- 
priate strategy for addressing both the internal and extei^nal issues, and 
that program review is. the essential core of that strategy. 

I would like to move now from the broader context and focus on more 
specific aspects of program review, including its purpose, some of the 
obstacles, and some matters of procedure. First, we should establish the 
principle that the basic purpose of reviewing existing programs, and ter- 
minating some of them, is not to save monsyi rather, ~ it is to maintain 
and enhance quality, vitality, and responsiveness during a time when 
large infusions of new resources through growth will not be forthcoming. 
It is a way living with less mopey through internal trade-offs, not 
necessarily a way of providing refunds to. the state. We must be very 
clear about this, because if we promise too much from program review, if 
we lead legislatures and state budget offices to expect immediate and 
substantial savings, we may buy one or two years of grace, but the long- 
term consequences will be disastrous. The faculty needs to understand 
that quality is an important objective of program review so that we have 
sonie hope of enlisting the endorsement and cooperation of the .professional 
community. Legislatures, state budgeters, and governors need to under- 
stand this too so that their expectations will not be unrealistic. 

Second, although the review, assessment, consolidation, and termina- 
tion of e}?isting programs is being forced upon us by external pressures, 
-it is an innately worthwhile activity, something we should have emphasized 
all along. Rather than an evil forced upon us, review is a necessary 
and beneficial activity to which, during a period when our energies have 
been directed toward expansion, we have given only token attention "and 
perfunctory performance. 

Third, program review is an extremely difficult process. It is dif- 
ficult because our colleges and universities and our systems (like most 
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social institutions) tend to be reluctant to embark on institutionalized 
self-examination. We don't like the kinds of questions ^that the people 
who review programs— even if our peers— tend to ask. It is also difficult 
because we lack the traditions and the management tools for this task. 
Finally, review is difficult because of complexities of internal governance. 
The collegial mechanism of an institution deals more confidently with issues 
of growth, where the primary conc.ern is dividing up an ever-expanding pie 
each year. Here consensual decisions come, more easily than when the issue 
is retrenchment. While the las't eight or nine years have produced a few 
very effective efforts at the institutional, system, and state levels in 
dealing with these difficulties, by and large, the rhetoric has far out- 
stripped the reality. We have lost precious time in establishing credible 
mechanisms that enable us to make this process work when the pressure is 
really on. I am reminded sometimes of the seer, Charlie Brown, who pointed 
out that "No problem is so big and complicated that i't cannot be run away 
from." Some of our reactions have clearly been along these lines. 

I am confident we can learn to deal with this era of retrenchment 
and constraint as successfully as we dealt with the era of growth. Those 
who persist and succeed will establish some basic structures and will 
learn much from their experience. But there is a certain, understandable 
queasiness about that first, perhaps long step into the unknown. 

Among the important contextual issues in program review which deserve 
special attention, one of the most obvious is that of institutional role 
and mission. We need to look at the way programs fit institutional roles 
and the ways institutional roles fit systemwide and statewide goals. 
Many of th^ issues here involve questions of quality, societal need, and 
the most effective use of resources. Questions will have to be raised 
Regarding the number of various types of programs needed and the appro- 
priate institutions in "which -to concentrate program resources. To make 
those determinations we must begin to reassess and in some cases define 
institutional roles and missions. The issues here include quality, < 
societal need, and the most effective use of resources. Immediately 
after the passage of Proposition 13 in California, Governor Brown appointed 
a blue ribbon citizens' committee, the Commission on Government Reform, 
which was chaired by Alan Post, the State's former Legislative Analyst. 
One of the Commission's recommendations was that California, having suc- 
ceeded in the early 1960s in establishing missions for each of its three 
major public systems, needs to move in the 1980s to establish differen- 
tiated functions for campuses within each' of those systems. I think the 
Commission was absolutely correct in that recommendation. 

A totally ad hoc^ cipproach to program review is to be avoided at all 
cost. The ground rules need to be made clear at both the institutional, 
system, arud state level, and must be established before the inevitable 
crises. His ti tutlons , systems, and states should establish credible 
processes which most of those concerned agree to in principle or at 
least understand clta^'ly before programs are reviewed or terminated. 
^^'^ '^d. hoc appr --='ch--wai ting for the crisis and then attempting to deal 
with substantive and procedural issues at the same time--is the antithesis 
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of orderly and thoughtful planning. It guarantees great resistance, be- 
cause it deprives the parties involved of that sense of "due process" 
which is absolutely critical in a consultative decision structure. 

Good data and analytical techniques are important, but mechanistic 
approaches must be avoided. There js. going to be a need, no matter how 
good our information or our techniques, to exercise judgment'. After allf 
the exercise of judgment is what leadership is all about, and we should 
not wish it any other way. A formula or set of criteria may be useful in 
identifying areas which need to be reviewed but not in making ultimate 
judgments about institutional programs. 

Interinstitutional cooperation at the regional level, whether it be 
interstate or intrastate, should be a part of program review. If we are 
to maintain an array of programs responsive to the needs of the people in 
our states, we cannot make planning and program review an isolated, insti- 
tutional exercise. We should coordinate program reviews within states 
and systems to determine whether cooperative arrangements for offering 
programs might be appropriate. We should plage greater emphasis upon 
interinstitutional arrangements, whether for sharing faculties or facil- 
ities, or for concurrent enrollment of students, or for reciprocity be- 
tween states and systems, or public and private institutions. It is 
essential that we seek out more imaginative and less costly means to 
maintain and eihance the service we provide. One way we can accomplish 
that is to bring institutions into closer alliances. 

My final observation is that those of us who work at the state level 
have a particular responsibi li ty— whether we are in state agencies, state 
legislatures or legislative staffs, or sf te budget offices— to be sure 
that, as we approach thi". period, we do not destroy incentives for insti- 
tutions to do those things that are most in the public interest. For 
instance, if all the resources freed by effective program review are im- 
mediately captured at some higher level, we destroy all incentive for 
leaders in our institutions to engage in wHit is really high-ri§^, 
high-conflict activit^^. If we cannot provide financial incentives in, a 
period like this, we can at least provide flexibility. We should help 
key decision makers in state government to understand that flexibility 
becomes more critical as resources become more limited. On that issue 
I am somewhat optimistic; in fact, I believe that this is a good environ- 
ment in which to ask for more flexibility'. As the funds become scarce 
and legislators and governors find themselves saying "no" five times , 
for every time they say "yes," the day may come when th^' wil 1 . happi ly. 
relinquisn their franchise on such decisions. If they have confidence 
that processes which are rigorous, humane, and credible are operating at. 
the institutional or system level, I believe they will tend to let these 
processes work. This does not mean that state governments should or 
will" adopt a totally "hands-off" posture. It means that we can work 
with some hope of success for constructive incremental change along the 
lines of greater management flexibility. When we are insensitive to this 
need, we may extinguish l^e behavior that we seek to foster in institutions 
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■ To conclude, the larger issue surrounding program review is that of 
self-definition and of conscious renewal of higher gducation in the years 
ahead. We know that no matter how well we plan, unanticipated events 
will overtake us" from time to time, whether they be recessions, wars, or 
Proposition 13s. But we should know by now that if we have some sense 
of purpose and direction, we can respond to events more rationally than 
if we drift from one situation to the next. Sometimes our greatest enemy 
is our own sense of impotence in dealing with our problems. There is a 
widespread feeling that any change is going to be a change for the worse 
and there is a tendency towards passivity with respect to the future. 
While we cannot control or sometimes even anticipate external events, 
neither can we abdicate our responsibility for responding, if not antici- 
pating, academic issues, and influencing, if not determining the academic 
future. 

I can visualize two possible scenarios for the decade ahead. In one 
we withdraw, become defensive, and refuse to face realistically the issues 
of planning and program review. The conclusion of that scenario is the 
erosion of quality and public confidence as higher education moves further 
and further to the periphery of American society and away from the center 
of life in most of our states. But there is a second scenario, one that 
calls for leadershi^j and a sense of purpose, for forthright recognition 
and anticipation of difficult problems, and for openness and rigor in 
attacking them. ^ 

The future is not, and has never beep, shaped by those who refuse to 
acknowledge reality or to make difficult decisions when required. It is 
not by avoiding tough issues that American higher education has become 
one of the major achievements of our society. Our response to the need 
for Planning and program review is but the next test of our vision, 
courage, ano leadership--and our commitment to quality. 

Thank you for listening. 
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STUDY OF PROGRAM REVIEW 

Robert Barak 'i^- ' 

For the last six months or so I've been serving as a consultant to 
NCHEMS on a study of academic/occupational program review both at the 
institutional and the state levels. The study itself had its formal 
inception last fall at a workshop for state-level academic /affairs officers 
in Seattle when^ there was some interest expressed in a study of scate level 
program review t.o follow up on and update one that Bob Berdahl and I did 
several years ago. There was also interest in finding out more about insti- 
tutional program review and approval; I want to share sotne preliminary 
findings with you. » J / . 

Before I get started, there, m^^be^eed to define a few terms. When 
I talk about program approval, T am r§Terring to the process by which new 
programs are proposed, examined, and approved. When I, talk about program 
, review, I am referring to the review of existing courS/fes. And to confuse 
you a little hit further, today I will be talking about both institutional 
program review/approval and statewide program review/approval. 

One of the key issues that we have identified is the problem of the 
impact of state agency reviews and approvals on institutional autonomy. In 
gcing around the country, I found this to be both a real issue and a fabrica- 
ted one. It is fabricated when institutions are n^erely responding in knee 
jerk fashion with outcries any time anyone outside the institution tries to 
influence the internal processes even if i t is a legitimate. exercise of author- 
ity. On the other hand, it is very real when state agencies overstep their 
legitimate interest and begin to poke around in matters which traditionally 
have been left to the institutions and their faculties and administrations. 
Perhaps it would be helpful if those of us who, have served in state agencies 
are aware of some of the intrusions on institutional autonomy. Examples of 
the problems that arise range from an incompetent bureaucratic state agency 
staff to excessive demands on the institutions by external agencies. For 
example, one university provost indicated tp me that his institution had 
just finished the process of a ten-year regional accreditation review. This 
consisted of exhaustive self-studies w-ithih the institution, numerous hours 

.of working faculty committees, students and stdff. During the same time 
period they had on campus nine professional accreditation evaluations taking 
place. These were also absorbing various amounts of administrative and faculty 
time.. Some of these evaluations were in the process of closing down, others 
were in the process of beginning. . .but they were all absorbing large amounts 

•of faculty and administrative time. This same institution was also a part of 
a state system and therefore the system office had program reviews that were 
'tIso being conducted. In addition, this institution was in a state which had 
a coordinating agency for postsecondary education that conducted program reviews 
Finally, the .provost noted that the president of the institution had recently 
p» fsed that internal program reviews be conducted. The provost concluded 
his v,omrient by saying that all of this review activity had to be counterproduc- 
tive. He said, "If anything, these reviews should show that we're doing a 
lousy job of teaching (our main purpose) because we're spending most of 
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our time doii.g program evaluations." Had this institution been in another 
state, two additional evaluations would have been taking place. One large 
eastern state has decided) that it Hill no longer approve a program for an 
indefinite period of time. From now on, every program is approved for one* 
year only. That same state's postsecondary coordinating agency had decided 
that it did not like the regional accreditation visits, and so it was going 
to do its own. So, institutions in that state also have state-level accredi- 
tation visits, which essentially accomplish the same objectives that the regional 
accreditation visits do. There may be some legitimate reasons for questioning 
the value of reg.ional accreditation, but it does seem to me that excessive 
demands for program evaluation are being made of the postsecondary institutions 
in this state. Somehow we need to reduce the extent of program review to more 
reasonable levels. 

A second isCue concerns our need to encourage good institutional manage- 
ment or, conversely, avoid discouraging good institutional management. An 
example of this issue was a situation in one state where an institution had 
done an e)^.nent job of internal program review resulting in the cutting back 
of a number of programs. In addition, this institutio'n had tied program re- ^ 
vi^w to their budgeting and planning p. icess. Unfortunately, the decision was 
later made the state level that resources of all institutions would be cut, 
by 10 percent across the board. What happened was a real discouragement to 
that institution to ever again want to do internal program reviews in a mean- 
ingful way. Several of their sister institutions had not done any internal 
reviews and so "the "fat" was still in place when they were told to cut 10 per- 
cent, while the institution that had been doing the good job of program review 
was actually penalized because the 10 percent cut went clear to the "bone." 
If anything will destroy meaningful reviews at the institutional level, across 
the' board cuts without regard to institutional inanagement efforts is certainly 
going to do it. 

Let me move on to another issue--the relationship of program review to 
other on-going management processes. When I think of program review, I like 
ideally to think of it as an integral part of the planning and budgeting 
process, but unfortunately that's not always the case. In one state, for 
example, I spent three hours with the people who were doing the reviews of 
dcademic programs throughout the state. I moved from that office to the 
person who was responsible for planning for the state (who, incidentally, was in 
the very next office), but that second person didn't even know what programs 
were being reviewed or what the outcomes of the previous reviews had been. The 
same thing, unfortunately, happens at the institutional level. I remember 
sitting in one office where the associate provost had a whole wall of note- 
book binders, each one representing a particular review of a program. He was 
very proud of them. I asked, "What do you do with the reports after you've 
comp.eted the review?" He stammered around a bit and finally admitted, 
"They just sit on the shelf." As I pressed further it appeared that there was 
no tie-in with budgeting and no one responsib'ie for planning at he institution 
ever utilized the reviews. There again, the process of review was completed, 
but it wasn't tied to any of the other management processes in the institution. 

Still another issue is the purpose for program rpview/approval . The 
purpose varies from institution to institution and state agency to state agency. 
The purpose for rpview at the various levels can be seen in terms of a continuum, 
with the institutional reviews at one end and. the r.tate reviews at the other. 
In an earlier publication I used the words "formative" and "summative" to 
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describe the differing emphases. Institutional review tends to be more 
formative while state-level reviews tend to be more summative. This is 
not to say that institutional review can't be summative or that state-level 
reviews can't be formative, but they are likely to be oriented that way. 

Almost universally, every institution representative I talked to said 
that the reason for their interna^ reviews was program improvement. The 
exceptions, though, were the institutions doing program review because of 
financial exigency. Cutting programs in this instance seems to take top 
priority, although it is noted that weak programs were being cut in order 
to maintain the stronger programs. 

Consumer protection was anothe>* purpose that was mentioned, usually 
as a function of the state agencies, especially those with responsibilities 
for institutional licensure. 

Planning was yet another purpose for program review. While I did f^nd 
some institutions using program reviews' for planning purposes, this purpose , 
was much more likely at the $ys4«gi and the state levels. 

In some institutions the purpose of program review was tied closely to 
budgeting. At the state level, though, this purpose seemed more of a goal or 
promise than a reality. The programs selected for review depended a lot on 
Ihe purpose of the review. At the state level there is the added cqnsjdera- 
hon of time and resources--how many programs can be looked at in a meaningful 
way given time and fiscal constraints. 

One last issue: One of the responses to our earlier study of program' 
review is that I get a lot of requests to work with people at both the insti- 
tutional and state levels on program review activities. One of the questions 
that inevitably comes in the beginning of our discussion is the. question of 
"who has the best system." The implication is that there is an ideal system 
out there that will work for everyone and then they v^ill plug that system into 
their own institution or state. I discourage this line. of reasoning whenever 
I can. It is important to find an apprbach that is geared to the environment 
and circumstances of an individual institution or state. I think it's a mistake 
for any institution to pick up ar program review/approval system from another 
and plug it in simply because it's been successful elsewhere. The same thing 
is true for states-; what works well in one state may be a disaster in another 
state. 

I think that drveloping a review process through broad constituent involve- 
ment is very important both at the institution and the state level. I was very 
impressed, for example, with one eastern school that used this approach in 
developing its program revi^ system. The wt^le developmental process began 
with a speech at the beginning of the year by the president of the institution 
who introdured the effort and explained its purposes and goals and encpuraged 
active faculty involvement. This started the faculty thinking along the 
linLS of program review. This was followed by periodic up-dates of the program 
review activity in the campus newspaper. Because it grew out of broad 
cimpus-bdsed involvement, the process was established with relative ease and 
a spirit of cniiwnitment by all involved. Initially the reviews were used to 
decide where the weaknesses of the institution were, and later when a financial 
crisis developed, it was actually used to, eliminate weak programs without morale 
problems. I think the merit of the process was that people agreed. on the process 
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before the crisis hit. Once a crisis hits, people don't think rationally 
and contemplate the various aspects of a sound program review system. 

The last issue I wanted to touch on was the political repercussions 
of program ^'i'scohti nuance. In almost every instance where program dis- 
continuance has occurred (save those programs which existed on paper only) 
there has been a political reaction. At SUNY at Albany it resulted in the 
reinstatement by legislative action of an Italian Studies program, tven pri- 
vate institutions are not ommune to political pressures such as the reaction 
to a prdposed program cut at the University of Pennsylvania, which is primarily 
a private institution. Somehow we need to develop ways of minimizing the 
political repercussions of program discontinuance. In this regard, I was glad 
to hear that the Southern Regional Educational Board (SREB) has a grant from the 
Ford Foundation to address the question of program discontinuance and institu- 
tional closure and t;o find ways to make those situations, when they occur, more 
palatable to the various internal and external constituencies. 
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PROGRAM REVIEW STUDY: AN INTRODUCTION 
Lilla Engdahl 

The study of program review which Bob Barak and I conducted Involved 
extensive surveys of and interviews with representatives of state higher 
education coordinating and governing agencies, colleges, universities, 
and system offices in the West. I have asked a few of these people to ' 
share with us today some of the approaches to program r-eview used In theiir 
states, as well as some of the issues which their experience has enabled! 
them to identify as important in the conduct of program Review at both 
the state and institutional levels. These are oil people intimately I 
acquainted with the problems and possibilities which program review i 
offers. 

The complete report of our study, including state-by-state profiles! 
of actual practices, will be published with tlje proceedings of this meeting 
I expect that report to answer most of your specific questions, but I will 
be pleased to answer any yo'j might have at this point. I am sure that our 
speakers will also respond to questions fonowipg their presentations. 
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HAWAII 
Durward Li ng 

The Unive/sity of Hawaii is a multi-campus institution consisting 
of: A major coinprshensive research university campus of 21 ,000 students 
(Manoa); a second complex that enroll" 3,000 students in four-year colleges 
of liberal arts and agriculture and a community college (UH-Hi1o); a new 
and very small jpper division program (West Oahu College); ana six com- 
munity colleges organize!;! as a subjnit. Each of these four major units 
is headed by a chancellor responsible to the president of the University. 
T'e University faculty and staff aro unionized and we bargain collectively, 
e» argetical ly and*"constantly (little about this relationship yet), 
i 

Hawaii is unique in that its higher 'Education budget is net enroll- 
ment driven, rbthsr, it is politically driven. Also, despite the legality 
of T ts Program Budgeting Lyrt^'ms, which was established with the help of 
John Keller, whom many of you .\now, it is more a form and a technical 
form at that, and the reality of program budgeting and/or performance 
budgeting is a far cry f ; om the literature about it. 

During the 196n's the University of Hawaii had an exceptional ex- 
perience in state funding. Behind the leadership of a governor who 
f Je higher education one of the top priorities of the new state' and its 
tLiture, the government provided generous financial support to the Univer- 
sity in budgets that more often thv-n not exceeded the University's request. 

This d1] changed in the early 70's and rather abruptly; in the 1971-73 ' 
bienniuni, the Univeri-ity ' s support vas drastically reduced by eliminating 
funds aii'i positions amounting to more than a hundred positions. (Appearance 
)f Grant Strategy. ) . - . 

Ainong the many redcttt)ns within the Umve»^3ity was one by the Manoa 
utHnic Senate which established a 'Program Review*' policy and procedure 
/ iuirint) th*^ review of each cicademic program every fiHh year. Shortly 
afterwcird, tW Board of Regents enacted a similar policy pertaining to 
th? review and evaluation of current and new programs in all campuses. 

The Board policy, approved February 8, 1973, required that each 
established inr»trL»ction» research, and public service program be reviewed 
every fifth yo.ir. It stipulated that "at a minimum, these reviews must 
consider pro^jrcifTt objectives, priorities, target groups, costs, funding, 
faculty, fdc i 1 it ICS, measures of effectiveness, 'and continuing need and 
iJf^niand," The policy delef)ated to each campus the responsibi 1 i^.y for 
devvOopifKj it^ own t'we-year program review schedule and fo^ suoniitting 
an up-dated version annually to the Office of the President. The reviews 
'.ompiMted i:?fK^ year are require'' to be submitted to the Office of the 
PreMdc^rtt in "the prescribed evaluation format." "Primary responsibility 
for ^\c»ttinq proqrani priorities, develofinq new program proposals, and 
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evaluating existing programs rests with the faculty, students, and 
administrators at each campus." 

In addition," proposals for new programs were required to be re- 
viewed' by the Board, and if approved, they become provisional programs 
to be evaluated at the end of the first complete cycle. No tenure \ 
decisions or commitments may be made during provisional status. \ 

The University Administration developed a program review format \ 
for new and established programs known ominously as PPB #10. The three' 
major units (our fourth unit began in 1976) developed their own interna? 
procedures and additional formats, and the process of systematic review 
began with the 1973-74 academic year, with the first reviews due in ^he 
President's Of/ice no later than June 30, 1974. In addition to the re- 
quirements of /the Board and the University Administration, each Unit 
established i'ts own procedures. The Faculty Senate and the Chancellor's 
Ofvfice of Manoa, our comprehensive University campus, required the 
Senate's involvement at the last review step prior to the Chancel lor'.s 
review. 

The 1971-73 fiscal and position reductions in the State's appro- 
priation had also caused the Manoa Faculty Senate to declare a morator- 
ium on new graduate programs. The year following, the University 
Regents, under pressure from the State Legislature alleging over- 
expansion of^gr.^duate and research*programs at the University, declared 
a moratorium on graduate programs pending a collective review of the 
^'dmini strati on of graduate programs at the Manoa campus, including the 
criteria for new programs. This review was conducted during 1975-76 
and the moratorium lifted by the Board in 1976. 

Another action taken by Manoa is also integral to this story. In 
1974, in consultation with the i^estern Association of Schools and Col- 
leges, the campus decided to link its accreditation review with the 

• program review. The decision included an agreement that the self-study 
aspect of the accreditation review would be carried out by the program 
reviews and that annual visiting connittees from WASC would focus" on 

' -'-e programs reviewed fdr the year of the visit. Manoa's schedule of 
leviews had attempted to link programs of similar nature or close inter- 
relationships for each annual review. 

The process originates at the department or program level, proceeds 
to the college level w.hen the review committee is supplemented by faculty 
from other colleges, then to the college dean, and then to the Senate 
Executive Committee who transmits to the Chancellor. The Unit Chancellor 
transmits a completed review to the Vice President for Academic Affairs 
wi'th -copies of his "closure" memo and, if Board action is required, a 
recommentiayion for that action. While research institutes do utilize 
teams of experts outside the flanoa campus and the State, t. instruc- 
tional programs do not. Neither is required or prohibited from using 
outside experts. The instructional programs have conside-ed the one 
expert 'on the accreditation visitation committee as sufficient. 



43 



In the community colleges, programs are reviewed at the level of 
concentration on majors offered in the associate degrees and at the 
divisio'.ial level for the general core requirements. The relatively 
short cime required to produce a graduating class has seemed to increase 
the opp.nrtunity for program adjustments in contrast to the University 
campus. A-^ter the first level o^' review, the community college campus 
provost (the chief campus administrative officer) makes an evaluation 
and transmits the document tq the Chancellor. The' faculty senates do 
not become involved in the process in the conmunity colleges, simplify- 
ing it somewhat. The Chancellor submits completed reviews to the Vice 
President for Academic Affairs who makes an annual report to the Board 
of Regents. 

The linkage of the review process to the allocation of resources 
^ is unstructured and incidental, sometimes almost as 1f by serendipity. 
But the uncovering of serious quality or accreditation problems leads 
to action! Me have inaugurated an Authority to Plan step and a "Special 
Study" status for prograqis indicating problems. 

The *Uni vers i-;y is now completing its fifth year of the program 
review process and is beginning an in-depth evaluation. We have re- 
viewed at some level over 200 programs, although only 120 have reached 
the Universitywide or system level of which eight programs were dis- 
continued. This is a 48% rate of on time completion of the cycle, 
although all began on time. I am pleased to share with you some of 
our tentative findings.. 

j My first observation is that the Manoa Faculty Senate really got 
out front on the issue of program review and then lost it. The admini- 
stration, tried to strengthen and support the initiative but somehow 
fa^iled to make it as effective or as positive as it might have become 
arid r.ianaged to create negative reactions. There are few Institutions to 
m^ knowledge where faculty organizations have themselves initiated struc- ■ 
tured review process beyond the department. Of course. It is also clear 
that by sejzing the initiative with the extended and laborious process 
the faculty protected its role in the review and probably minimized the 
use of review lor, fiscal decisions. , 

■ The second observation is that the mixture of purposes and objec- 
tives in the establishnient and use of the reviev/s comprpmised their 
value the institution. The origin of the review policy and process 
at the Tlagship campus seemed to reside in a desire among key faculty 
leaders and the campus administration to make appropriate program ad- 
justnents (especially contraction, discontinuation, or moratoria) in 
a temporary fiscal crisis and to put in place a process by which future 
'adjustments would be made if the fiscal crisis continued. The Regents' 
and Administration's objective seems to have been similar at the outset 
although it is clear that the ]3oard firmly believed that there were weak 
or high cost and low priority programs that should be di-,t;ontinued as a 
result of the review and that the sunset approach ^u1d be applied. P^erhaps 
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it is fair to say, however, that most of the faculty quickly/ietermined 
to use the program review process for diagnostic and improvement purposes 
only and senatof^ial courtesy prevaiUd. The Regents gravitated to the 
position that the review process was intended primarily to target pro- 
grams that should be discontinued. A review of the history, files, and 
current impressions reveals that the behavioral responses had not been 
calculated carefully enough for step's to be tak6n to increase the prob- 
ability of positive implementation/ For example, the behavior of the 
faculty, over time has come to reflect their view that the entire process 
(which was initiated by the Manoa faculty) is simply another ritual that • 
serves the bureaucracy, especiarlly the farther removed r levels of admin- ^ 

i strati on and the Board. > 

/ 

/ 

Third, "Program" was not adequately deffned and as a result, it 
was transla-ted "organizatibnal subunit" by some campuses, and degree 
majors at others, to create a bit of confusion. Prior to the Review 
policy, there was not a stated policy on Regents authorization of pro- 
grams. Nor was there a real Academic Plan. v-. 

A fourth finding is that the process is so stretched out in terms 
of multiple reviews that it is unclear which review l^vel is responsible 
for what. Therefore, each level simply transmits it to the next. 

Fifth, at the policy level, theCe is an absence of criteria to be 
applied in the program reviews. While need, value, quality, and prior- 
ity are mentioned in the rhetoric, they are not stated in operational 
terms. Succeeding levels have focused primarily on the ouality of the 
program being reviewed. Staff planning for personnel reallocation, 
rearrangement, and renewal needs to be included. 

Sixth, the consp.icuous absence of a review policy, for administrative 
and support functions, organizations, and programs' has contributed to the 
notion that instructional programs are special ta.rgets of rev^'ew for the 
• purposes of discontinuation. We already focus too much analysis on 
instructional inputs on our institutions. 

Seventh, while the various academic* plans of the campus have been 
influenced and tempered by inferences growing out of program review, 
the review process has not served to establish priorities in program 
development. It has permitted short-term decision/shifts toward long- 
tern objectives.' 

Eighth, examining prd(grams by a one-fifth slice on an annual base 
or a two-fifths cluster for a biennial term has not contributed toward 
evaluations by comparability or as a campus-wide entity such as a self- 
study would provide. ^ \ j . ' 

Ninth, program review may actually obscure priorities and realloca- 
tions (as PPBs emphasize data on the expense cf policy). In linking it 
to budgets we have little experience and understanding in predicting 
trade-offs. 

.'7 
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•Tenth, the process, while logical, is very costly in the expendi- 
ture of phychological and physical energy and contributes to constant 
uncertainty. ^ 

Eleventh, the use of the annual reviews in the accrediting process 
is also logical and promises to conserve energy, but it has changed 
fundamentally the accrediting function, and in my opinion, has served 
an unintended substitute for an all-campus review. Moreover, it has 
proven much more expensive 1n terms of fiscal costs, coordinating duties 
and report processing. I would not recommend an Identical repeat of the 
method we' ve used. 

Twelfth, the "classes" of action to be taken as a result of the 
program review need to be clearly known by all participants. At the 
moment, \n our program review, t^e imagery is that one of two primary 
actions is expected, to continue or to. discontinue the program. Of 
course, a number of other less "fijial" actions are taken by the vari- 
ous participants in the process but*since the range of actions Hkely 
at diffet<ent levels is unknov/n, the reviews are quite defensive. 

A thirteenth major finding is that our present review is based so 
much on un-it and subunit evaluations that there is almost no opportun- 
ity to look at curricula needs and programs across the system or even, 
across the campus. Health sciences,\agri cul ture, etc. never 'come 
together. , 1 

1 \ 

Our fourteenth finding is that bro^ram review and academic planning 
have to be. linked to the sources outs\ide\Of the institution to be viable 
and effective. Program review is really a. political process; we're 
going to havo to become accustomed to involving people outside the insti 
tution in that process. I know that sounds like capitulation,' but the 
kinds of programs we offer in a public university may be a matter of 
public policy. This is thf. point I want to close on; We've learned a 
great deal from program review. We have now come to the conclusion that 
we really need two kinds--one for the department to do on its own, and 
another, a quantitative review that identifies certain peril points in 
the program. And at that point, we may -need to use consultant teams, 
someone who has no vested interest in the outcome of the review, to 
• make qualitative judgments. 

Our conclusions at this time are: 

1) It is essential to articulate clearly why we do what we 
do; 

2) The policy for review initiated by the faculty was a healthy 
positive step; faculty support and understanding should be 
the basis of any program review; 

3) The reviews have influenced attitudes and behavioral patterns 
for which insufficient thought and planning were given; be- 
havioral objectives must be thoroughly anticipated; 
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4) The reviews have probably denied campus-wide or unit-wide and 
certainly university-wide se;if-studies and priorities; 

5) Much of the" data section is left to the department or program 
to locate, use; University Adraini strati on needs to provide 
off] cia'l data to each program; 

6) We must develop a simpler process that includes "branching," 
based on certain findings, uses outside expertise, and one 
that states more quantitative criteria as reflective of a 
peril or quality indicator. 

7) Ue are novi considering ways to initiate and institutionalize 
flexibility in program completion/composition; 

8) We are, as a result of all this, considering radical change 
in program review; quantitative peril point campus review 
for quality. 

We hope that our experience will be helpful to other institutions 
developing new systematic program review systems. 
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' CALIFORNIA COMMUNITY COLLEGES 

John D. Meyer 

The California Community College system ineluc^es district organiza- 
tions ranging in size from a single college to nine. 

At the state >evel the Chancellor's office is involved in program 
review prior to program implementation. We published a handbook on pro- 
gram review procedures two years ago. Because of the existence of both 
multi-campus and single campus districts, we often face the question of 
local autonomy. The history of the development of the California commu- 
nity colleges is one in which the district and the local institutions have 
jealously guarded their local autonomy. In" fact, the law under which we 
carry on the program review activities stipulates that we must make every 
effort to preserve local autonomy, and we are reminded of this whenever 
we suggest to an institution that it should not offer a certain program. 

All community colleges in California are comprehensive, which means 
that they are involved in vocational education. In the past this has 
meant that Vocational Education Act funds have been available to the col- 
leges for certain kinds of research projects. One of these projects has 
resulted in a program review process which has come to be knoWn as tlie 
Community* College Occupational Program Evaluation System (COPES) and is 
the major thrust of my comments here. COPES started in 1971 — long before 
Proposition 13. It is a qualitative' review process for programs and in- 
volves advisory committees composed of all segments of the population 
including industry leaders, program graduates, and present students in 
the program. First, the faculty, students, and administrators are asked 
to fill out a questionnaire which asks a response to a set of qualitative 
evaluation questions. The answers are summarized through the use of a 
computer at Foothill Community College, and the printout is given to a 
visiting team made up of people who know about the subject matter of the 
program being reviewed. This team validates the results of a self-study 
prepared by faculty and administrators from the program by contacting 
the people who contributed the questionnaire answers. In 1977 only the 
Home Economics and the Administration of Justice programs were reviewed. 
At t^ t time, eight colleges from each of these programs were selected 
around the state and each program reviewed in depth at those places. The 
requirements of the Vocational Education Act stipulate that evaluation is- 
a condition of receiving further federal funds, so our design was to use 
this pl^ocess to satisfy that requirement. 

In an effort to expand and improve the use of COPES we have divided 
the J06 colleges into four stratified samples and established through 
„ statistical techniques that each of these samples is representative of 
the entire system. In this plan, every four years each of those colleges 
would go through a review of its entire occupational program. Special 
interest areas such as "administration of justice" could be reviewed on 
an ad hoc basis. 
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Proposition 13 had a detrimental effect on our intent to embark on 
this scheme because it reduced our staff and funds. We' had planned to do 
a random sample out of each' of the stratified samples in order to validate 
the process, but funding cutbacks meant that we only visited one college. 
We did experiment with that one visit this past year by tying it to the 
previously scheduled accreditation visit. The results of this visit are 
not all in yet. A great many reporting activities were demanded of that 
college at that time, however, including a Department of Finance requirement 
all within about six weeks--and so the test was not as good as we had hoped 
it would be. Our nope is that we will be given the necessary funding to 
expand our activities systemwide for the next fiscal year. 

In addition, we have another Vocational Education Act program review 
process as the result of, the development of two books called Guidel ines 
for Occupational Planning . These are handbooks for planning and develop- 
ing programs. They also take up the questions of how' programs might be 
discontinued. (These have not been top sellers among the col leges, " but 
we have them available, anyway. ) Proposition 13 has had another periph- 
eral effect or. this area. The major districts, Los Angeles in particular, 
are beginning a rather inte' sive program review on their own in which they 
are looking at quantitative data— the, number of students enrolled, number 
of graduates, and so on. We are "trying to keep the COPES concept separate 
from this quantitative evaluation, though the discontinuance of programs 
necessitates' looking at both qualitative and quantitative data together. 
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ARIZONA 
William Phillips 

• 

A few months ago when I was asked to serve on the panel at this 
workshop, I agreed, thinking that I would be expected to present a truly 
rrodel practice of system-wide academic program review. After spending 
four years in this sort of activity in another state, I had come to 
Arizona and had been able to designand develop and pLt into place a 
completely new academic program review system which I had hoped would 
draw upon all the good practices and avoid all the pitfalls which I 
had discovered through personal experience and communications with 
colleagues in other states. 

I do have such a model of a system-wide program review which I would 
be pleased to share with any of you who would like to read it. Briefly 
the Arizona system proceeds along two separate lines: each institution 
is mandated to conduct internal reviews of all academic programs over a 
ten-year per,iod. While the reviews are conducted by the. institutions 
involved, it has been my experience that colleagues in other departments 
are likely to be as critical of the program quality of someone else in a 
university as any outside critic who might be brought in. Thus, these 
reviews do not tend to be entirely self-serving, as might at first be 
imagined,. The results of these reviews are presented to the Regents' 
Central Office as each review is completed. 

The system-wide phase of this activity is conducted on an^ "as indi- 
cated" basis. and it focuses on duplicative programs which may be flagged 
for system-wide review upon significant deviation from regional and na- 
tional norms or other indicators taken from a list of 15 such indicators 
for identifying programs for system-wide review. Once an area has been 
selected for system-wide review the universities involved pirepare lists 
of 'outside consultants whom they would consider completely acceptable. 
All of the names submitted are put on a single list and then each univer- 
sity is permitted to ask that any consultant be remove.d for cause. Tho^e 
consultants whose names remain on the .list are obvnously acceptable to 
all of the institutions and it is from this group the final consulting 
team will be chosen by the Central Office. Prior to each consultant's 
visit each institution compiles data and other information in an agreed 
upon format. The consultants are then brought in to look at the programs. 
They write a report that is reviewed by the respective institution's for 
factual errors before it is shared with the Regents. Eventually the 
report goes to the full Board. 

The consultants are asked to look at the quality of the programs, 
the resources available to the programs, the outcomes of the programs, 
the program costs, and other factors. The consultants visit with univer- 
sity administrators, faculty, students, persons from the community, and 
representatives of any professional group that would be involved with 
the employment of graduates of the program under review. Finally the 
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Board of Regents, after reviewing the consul teNits ' reports and recommenda- 
tions from the Central Office, may take actionsXto leave programs in' the 
status quo , establish new programs, strengthen existing programs, modify 
existing programs, or phase out or discontinue existing programs. 

This is the system of a system-wide review which we have in place. 
I could say a lot more things about this model, and v^ll be pleased to 
discuss it with anyone who is interested. Although w^have not completed 
a full cycle of this activity in Arizona, I can predict\with great con- 
fidence that it will work about as well as these processies work in other 
states. That is to say, there will be a lot of activity\t both the 
.institutions and in the Central Office; the end result wilNj most likely 
be that few programs will be terminated, and those which ate will ba 
quickly replaced by something equally as expensive at the institution 
that lost the program. 

With the foregoing in mind, today I would like to describe a differ- 
ent approach to program review which we are in the process of carrying 
to its completion in Arizona at the present time. This procesr is based 
upon the assumption that the internal, academic questions in program re- 
view are much more easily answered than the external, political questions. 
Therefore, instead of having a team of academics review programs to in- 
vestigate the academic quality of the activity, we began with a process 
intended first to defuse the external issues. 

Before proceeding further, I should make certain that you understand 
the background of the higher educational situation in Arizona. Arizona 
has a population of ' approximately two and a third million people who are 
served by just three public institutions of higher education. These three 
universities enroll 77,000 students on a head-coint basis, about 63,500 FTE. 
There is only one private, accredited, four-year institution in the state— 
a church-related school with an enrollment of about 1,300, and one accred- 
ited upper level, non-traditional proprietary institution with an enroll- 
ment of approximately 200. There are severalnon-accredited institutions 
and several out-of-'-.tate operations active in Arizona. More than half 
of the state's population is located in Maricopa County, the home of 
Arizona State University, and approximately 25% of the state's population 
is in Pima County, the home of the University of Arizona. In many respects, 
Arizona is one of the most urbanized states in the nation, with more than 
75" of the population residing in major metropolitan areas. 

The program review in question concerns agricultural programs and 
was brought about by the fact that agricultural programs are being offered 
at both the State's land-grant institution, and another university having 
B Division of Agriculture located in the College of Engineering. The non- 
land-grant university dots not operate an extension service,' but it does 
have an experimental farm and, to all intents and purposes, the public 
can tell little difference between the activities of the two agriculture 
procjrams. The existence of the duplicative agricultural programs has 
been the subject of considerable interest by the Board of Regents, a 
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constitutional governing' body with, jurisdiction over all three public 
four year institutions, for a number of years. As recently as 1971 a 
committee had been appointed to investigate the situation. 

It was against this background that in November of 1978 the Regents 
appointed a seven-member comnittee,, composed of two Regents who had 
agricultural interests and five prominent agriculturists from the state. 
The five lay members of this group hold the key to whatever success this 
effort will ultimately enjoy, for 'they represent all of the major agri- 
cultural commodities grown in the state as well as all agricultural re- 
gions of the state. They include men of impeccable credentials: a former 
president of the State Senate, a former president of the Mi-Ik Producers 
^ Association, a former president of the Beef Cattle Association, a former 
president of Cotton. Producers,- etc. All five individuals are men of con- 
siderable expertise and reputation in agricultural circles, as well as 
having influence throughout the state. The Committee met, with the 
Regents' academic officer serving as Secretary of the Committee, for six 
full days spread over a five-month period. . During its investigations, 
the Committee visited both universities involved, meeting with administra- 
tors, faculty, and students and visited all of the university farm proper- 
ties located throughout the state. Two parts of the final report, all 
of which was unanimously agreed to by the Committee, dealt with the 
experimental farm properties and the agricultural extension service. 
' The third part of the recommendations, in which you might be interested 
today, dealt with academic programs. The Committee reached the conclu- 
sion that the academic programs operated by the two institutions are 
unnecessarily duplicative, and recommended the disestablishment of the 
division of agriculture at the non-land-grant university. 

The conclusion was neither novel nor unexpected. Indeed, almost 
any group which might have been brought in from outside the state could 
have easily recomnended the same action. The programs recommended for 
discontinuance are more expensive than those at the competing institu- 
\ tion, lack the resources, and simply do not have the potential for ex- 
\^cellence of similar programs at the land-grant institution. The Comiiittee 
^did recommend that the adversely aff,ected institution should concentrate 
on agribusiness pvograms but that thfese should be based in the College 
of Business Administration, not a Di\lisiori of Agriculture. The Comnittee 
was also willing to leave agricultural engineering programs, which were 
already integrated into the school of engineering, in place. 

Unfortunately, I cannot give you the final outcom? of this scenario. 
The Board received the report at its May 25 meeting, which happened to 
be at the university whose program was recommended for termination and 
was confronted by- a large and fairly hostile crowd of faculty and students 
who felt that they had something to lose by the adoption of the recommenda- 
tions of the report. 

None of you who have participated in a review of an academic program 
which results in a recommendation for discontinuance will be surprised 
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to learn that the report, which was initially distributed only t Regents 
and University Presidents, was somehow leaked to other groups and widely 
distributed, or that a number of letters seeking support from the public 
were mailed using the postage meter of the university which might be 
adversely affected. In deference to the large crowd, the Regents have 
scheduled a public hearing on the subject in order to give everyone who 
is interested an opportunity to speak. It is anticipated, however, that 
action will be taken no later than the July meeting of the Board. While 
I would be extremely fcjlish to predict the vote of the Board on any issue, 
it is important to note that the governor and both of the Phoenix, newspapers 
have spoken strongly in favor of adopting the recommendations of the report. 
It would be surprising if most of the Committee's recommendations were not 
adopted. 

I wasn't joking, then, when I tok you that this was an example of 
program review from Z to A. The political issue has been tackled first, 
on the assumption that there is no point in doing the academic wojjk of 
program review if the politics of the matter will preclude or thwart any 
actions. If ,the hard decision is made to discontinue the program then a 
great deal of work must be done by the university, with the guidance of 
the Central Office, in determining how best to deal with the program 
pieces which may be left and how to handle the discontinuance. This is 
an especially critical issue. since the Committee has recommended that a 
part of the program— agribusiness— be retained at the university but be 
shifted to another college. I think the significant thing about this 
review process, however, has been that, while the conclusion of the 
Committee was obvious, and perhaps inescapable, had this conclusion been 
reached by a group of visiting academic experts, there is no doubt in 
anyone's mind that thfe adversely affected university would have countered 
with recommendations made by its own experts, no doubt, equally prestig- 
ious, who would have recommended to the contrary. The possible result 
would have been an impasse between experts, and it is likely that the 
Board, facing the political realities of making an unpopular decision, 
would have found ample justification for doing .little or nothing at all. 
Instead, this process has gone to the heart of the matter by coming to 
grips with the really tough question— the critical one of whether or not 
there should be two agricultural programs--with the assumption that once 
this has been dealt with^ the academic process may proceed along a more 
rational and unhurried course. 

I regret having to give you so incomplete a report and will be 
pleased to bring any of you up-to-date on the final outcome of these 
actions in July, or at whenevisr time the Regents take their final action. 
However, I believe- that several conclusions can already.be drawn from 
this model of program. review. First is that the political Decisions 
are the ones which are really difficult to make. Even a constitutional 
Board of Regents whose members are appointed to eight-year terms is not 
insensitive to personal and political pressures. Indeed, the very proc- 
ess by which Regents are appointed, i.e., nominated by the Governor, 
confirmed by the Senate, insures that they are a part of the political 
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scene; Second is that while any action which the governing body attempts 
to impose upon a university is likely to be vigorously resisted by the 
university, this action is more likely to be successful if the Board tan 
feel that it has a strong basis for that actiofr and. defenses from outside 
the educational system. Third, and perhaps the^most significant', is that 
the academic considerations may well profit from their taking place after 
policy issues have been dealt with and disposed of. Of course, this 
model will only work with those programs which are professional in nature, 
or have any employee group that is clearly identifiable. The next pro- 
grams targeted for this treatment^ incidentally, are Schools of Architec- 
ture and a School of Mines. I sh"all be pleased to attempt to answer any 
questions which you may have about the process I've described. 
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Pmm REVIEW IN POSTSECONDARY EDUCATION; 
A RETURN TO JUDGMENT* 

Edward F. Kel.ly and Thomas Johnston 




Introduction 



The processes of program review, whether viewed at the institutional 
or statewide level, are analogous to the activities of educational program 
evaluation. As a result, program reviewers will confront some of the same 
issues and problems that have become the truck and baggage of t^e program 
evaluator. We argue that, like the activity of program evaluation,- the 
process program review essentially directed at the development of 
publicly justifiable judgments of worth. These judgments fbrm the prenpses 
of an argument that is made with respect to a specific educational program. 
The argument is principally a practical one, one that ledds not to new 
knowledge but to a decision to act with respect to an educational program, 
^ith the context defined as that of the practical argument, we will discuss 
the process of program review and its relationships to educational evalua- 
tion. Having offered some arguments on 'definitional similarities, we 
Jiscuss in turn the issues that program review will confront and then 
we suggest several alternative replies to these issues. This paper conclude'' 
with a set of questions that we believe will benefit the activity of program 
reviv t,. ■ ■ 

Lvdluatio.n Defined 

efinitions of evaluation tend to be varidus and at times conflicting 
(WortK'^n and Sanders, 1973), but there appear to be at least three facets 
that sjch efinitions share. Evaluation is directed at the detemination of 
worth iScr ven, 1967/. Evaluation is intended to assist decision making* 
(Stufflebeani, 1969),. and eval'Ji^ion is quintessential ly a political activity 
(House, 1974, 1978). In t his paper we tarke the view that evaluation is a 
fgglbojjplpgical activity inte nded to develop descriptions of programs and 
judf^ertts of wor th to fa cili tate dec i sion making . 

Building descriptions of (educational programs requires that evaluators 
constiurt lo'jical and empirical renditions of the conditions, activities, 
and outcome.; of programs (Stake, 1967). One hoped-for effect of this activity 
15 that people who do not participate in a program, but who may be required 
to m&kG decisions about it, will have a reasonable basis on which to know 
what it must have been like to be there. In a similar fashion, the require- 
ment that evaluation be directed at the development of judgments of worth 
(Scriven, 1967) entails the collection and weighting of evidence about the 

p'Jper wds prepared for presentation at the WICHE-NCHEMS Workshop 
ut ST.<;«'t?wide and Insti tutioaal f cstsecondary Educdt'on Program Review, June 
•5 b, ]9/'>, Sheroton Airport Inn, Millbrae, California, 
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worth of a program from one or several points of view. These points of view 
generally represent different steidards against which the program may be 
judged, and consequently, perceptions of program quality and adequacy 
frequently vary depending^ i>pon whose standards cire used for judgment. In 
response to this multiplicity, Stake (1967) argued that evaluators ought 
to be in the business of collecting and reporting the judgments of others, 
while Scriven (1967) argued that evaluators would, by definition, be 
required to render a specific judgment of worth, 

' The argument that Evaluating is best understood as a service system 
to decision making is well represented by the works of Daniel Stufflebeam 
(1971) and Marcia Guttentag (1975). Stufflebeam defined evaluation as the 
"process of delineating, obtaining, and providing useful information for 
judging decision alternative"" (p, 129). In an excellent analysis, he 
argued that decision makers at the local, state, and federal level most 
often require different types of information and that the same data will 
not usually^mest the needs of decision makers at different organizational' 
.levels. For example, inforir^tion that may ut: important vo. local program 
people as they attempt to improve their programs (wh?t Scriven [1967] called 
formative evaluation) is usually uf little use to st^te level personnel 
attempting to decide whether a program or project should be re-funded or 
accredited. 

Eva luation as a Practical Argu ment 

It is at'.this juncture, the relationship between information and 
decision making, that the activity of evaluating is best understood as a 
consideration related to policy analysis, policy research, and policy 
formulation. It is also at this point that the applicability of^the 
practical argument as a framework for understanding evaluation is best 
presented. 

When we speak of the practical argument, we mean a logical argument 
that employs a set of premises that are filled with val uational. and belief 
claims, premises that support a conclusion to act. In one respect, when 
we give r'easons that are intended tavjustify a decision that we have made, 
we are articulating the premises of tne practical argument that led to that 
decision. The activity of discovering and exposing the premises of the 
practical arguments in the minds of decision makers is a primary activity 
of program evaluators, one which they must become more sophisticated at • 
conducting. Program evaluators need to be mory sophisticated at doing this, 
because the premises of these arguments are frequently incomplete or not 
clearly stated. Furthermore, all exposed premises are not necessaril/ true, 
and false premises need to be detected since they bear heavily against the 
acceptability of the conclusions of such arguments. The process of id entifying 
the promises_of_ the practical arg ument and tlience of publicly portr aying them 
is tho h;?art of the_m_atter of^i)ro^ra£ 

One of the oldest an<l most traditiona' models of evaluation, ^he 
j>f<j'e(l i ta tion model, is a prime examp' the relationship between evalua- 
tion and ^ho practical arqument in a <i ision context. Glass ( 1974) identified 
thr dfwelopment of three major models of evaluation and. argued that the 
.V ^rodi tri tion niodnl was thp oldest of the fornidV eval uation processes dating 
I t to ihe early 1900's. 
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In addition, in a recent and highly important analysis, House (1978) 
has shown how evali/ation is possessed of a logic of argument and how, 
further, the warrant of any evAjuative activity is that it persuade the 
client or decision maker of the credibility of the evaluative claim. This 
activity of developing persuasive evaluations is a. highly political process, 
because it is intended to, justify the selection of jone person's or group's 
preferences as those o.f society at large (Riker and Ordeshook, 1973). This 
process requires a sensitive professional response to the varyi.ng beliefs 
and value dispositions that represent the groups that participate in, and 
are affected by, any program evaluation. 

Implicati ons of Defining Evaluatron as a Practical Argument 

To say that evaluation requires the determination of the worth of a 
program means that we must first believe that it is reasonable to argue 
from empirical evidence about a value laden premise. In a fashion 
similar to that argued by John Dewey in his Theory of Valuation (1939), 
the implication is that it is reasonable to believe that the goodness»of 
a program, a value laden consideration, can be empirically verified'. 

A second implication is -that there is a discernable -level of rational- 
ity in our program decision making, and that the discernable level is 
efficient. Inefficient rationality means that the correlation between the 
quality and quantity of information that is available and the excellence 
of decisions that are made is much less than perfect. Some of this lack 
of correlation probably results from a failure that has typified many 
evaluations: failure \o attend to the differing belief systems that their 
clients held (cf. 'House, 1978). 

The third, and possibly the most important implication of this posture 
L^-A!]jj_ev3.^uatio'^ as an activity will maximize one set of social values at 
the expense of another. "This process is important to understand and recognize 
for it, 'too, lies at the heart of the matter'of both public policy formulation 
and program evaluation. It is the political ordering of social preferences^ 
^J5-lL J^,llg_£IM.rani e val uator needs to be sophisticated at developing, and it 
w jj ] be the same orde ring tha t the processes of program reView, whether 
vj-g^ieiLj t th e local or the statewide level , .will have' to be effective a t 
'^_^_t^J^jLi'^i^ • Program review, like program evaluation, must be understood as 
a political process, intended to affect policy with respect to social pro- 
nrdins, institutions, and priorities. 

VJhdt Counts aji^ an Evaluation? Some Comparisons 

As d result of these definitions and this analysis, it follows that a 
number of other activities that frequently appear in social ^nd educational 
proqrain analy is must be understood as essentially different ^rom the activity 
of evdludtinn. For example, the program audit, a process intended to examine 
^np rolat ionship between the expenditure of funds and acceptable budgetcTy 
I dteqorizdt ions, while being a usef I accounting process when well conducted, 
is not an evaluation because it does not intend to document the worth of the 
I'fO'irMn I. ofK er'ned . 
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Similarly, other important developmental and descriptive activities 
such as survey research, operations management, and organizational develop- 
■ne^t, . 'though they perform important functions, are not to be confused with 
progran, evaluation. Although .organizational development efforts, for 
example, may utilize evaluation studies, they are not themselves evaluation 
efforts because they do not' intend to develop descriptions and judgments of 
program worth that are intended to facilitate decision making. 

Other well-known activities such -as cost effectiveness analysis, cost 
efficiency considerations, and co^t benefit studies are usually not acceptable 
as evaluations unless they overtly claim to present descriptions and judgments 
of program worth. Ellis Page's (1972) argument about the need and desi>^a- 
tiility of a standard measure of educational benefit which he caUed a Bentee 
may be an example of a bridge work between cost benefit analysis and program 
evaluation properly combined. Unfortunately, evaluators "have not been able 
to find a soc ial ly ^acceptabl e benefit standard .score for educational programs. 
Curiously, program review processes may imply one in that they require a 
judgment of the social benefit of an educational program! - 



Efforts to structure pr<igram e.xpenditures more effectively and 
efficiently, such as program planning and budgeting systems (PPBS) and 
the use of zero-based budgeting models, are not evaluations for the same 
reasons as described above. They are eminently political processes, just 
as evaluation is (cf. Wildavsky's The^ Politics of the Budgetary Process , • 
1974), but they are not evaluations. 

I n what follows we make the assumption that the activity of pro gram 
revi ev^ (PR) is ultimately intended to establish the worth of postsecondarj^ 
educational programs and that the information^ that is gathered under the 
aegi s of PR is in tended to inform and 'infl uenqe decision in the public 

Given that this assumption is true, we argue that PR means Program 
Evaluation (PE). Thus, PR = PE, and as a result, the considerations of PE 
should be used to frame any PR effort., - 

If Program Review Equals Program Evaluation, How Should It Be Done? , 

We begin by asking two important questions about PR, questions that 
are often asked about the activity of program evaluation. These questions 
are: (1) How reasonable is "t to conduct program revievi^? (2) How realistic 
i,s it to conduct program reviews? 

(^) How reasonable is it to conduct program reviews ? The reasonabi 1 ity 
of program review -is best determined through an examination of its goals. 
Consider four avowed yo Is , for example, that program review frequently claims. 
They are: (1) to eliminate program duplication, (2) to scrutinize programs 
thouqht to be underrproductive, (3) to insure consistency between academic 
pn gram goals and institutional mission, and (4) to determine the efficiency 
and effectiveness of academic program operations. 

These are not uncommon goals. They bear some correspondence to those 
that describe 'the outcomes of the current doctoral program review being 
conductefl in the State of New York. (Cf. Meeting the Ne eds of Doctoral 
fduration, Position F'aper 19 by the Regents of the University of Che State 
of New Vork . ) 
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One of the first questions that an evaluator might ask of PR is 
whether these are worthy goals. Are they obtainable, and are they deserving 
of pursuit in the first place? As we sugge'sted earlier, the worth of social 
objects frequently depends on whom you ask. Why they believe so is a con- 
sideration in argument and persuasion over values. 

From -ne state's view the value of PR may be argued to be that the 
state has a responsibility to monitor and improve educational services, 
and that, where necessary, this responsibility will entail strengthening 
some programs at the expense of others that are less worthy. It will mean 
establishing a process that is more useful than it is burdensome, a process 
that yields greater benefits to the social good than it incurs costs. ■ 
These goals will also entail a cooperative relationship between institu- . 
tutional representatives and program review operators, lest an adversary 
proceeding be inadvertantly established. Insofar asPR is understood as 
an application of program evaluation, then we argue aft^r Scriven (1974) 
that the cost of doing PR should not exceed thp worth of the information 

that the process yields. ' v 

>■ * 

The rationale for pro grap review needs to be stated clearly and - 
examined publicly. If this Ts not accomplished, PR will possibly be 
construed by some as an effort on the part of the state to gain further 
control of postsecondary educational programs. Others will view PR as 
a way to place more effective and efficient controls on the utilization 
of educational resources. Some will see PR as the death knell of academic 
freedom and may argue that it is a process intended to standardize the 
higher educational curricu'lum^. and thus stifle creative alternatives and 
new approaches. Therefore, one of the -first matters that needs to*'be 
considered is the rationale/or argument that gives reasonabil ity to the 
program review activity. c ^ ' 

Whatever these arguments may be, whether' about means or ends, it can 
be anticipated that there will be counterclaims and difficult questions 
that will be asked about the PR activity If such questions arise in PR 
as they do during program evaluation, they will be questions like the 
following: 

[1] How will program review discriminate reliably between duplications 
of programs wnen both programs appear to ■;be excellent bq^es? 

[2] How will productivity be defined, and how will the opportunity • 
for i'nnovation and potential failure be balanced against the 
productivity metaphor? 

I 

[3] How will program review avoid requiring educational homogeneity 
as a replacement ,for diversity? 

[4] How will program review discriminate between educational 
efficiency and educational effectiveness? 

Questions like these will more than likely be leveled at any statewide 
program review process. They deserve careful consideration befire such 
programs are implemented. 
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. (2) How r ealistic is it to conduct program reviews? If the activity 
of program review can be p'=>rsuasi vely argued as a reasonable activity, the 
second consideration questions whether the activity Is realistically intended. 
A realistic process for program review will require attention to three impor- 
tant aspects which are: '[1] the determination of criteria, standards, and 
indicatorb [2] operational definitions of critical constructs, and [3] the 
specifications of client, audience, and sponsored relationships. 



If the program evaluation parallel holds true, the following terms will 
need definition: criteria , standards , and indicators . By criteria we mean, 
the characteristics of programs that are believed to be important in pro- 
ducing a judgment of worth. Productivity, for example, might' be judged to be 
one criterion for program worth. Standards,, on the other hand, are the 
scales against which programs are compared with respect to a given criterion. 
Thus, if productivity were a criterion, a standard would identify the appro- 
priate level of productivity that would discriminate among superior, acceptable, 
and unacceptable programs. Indicators are evidence that is believed, by 
somebody, to be representativ»e of the criterion concerned. An indicator of 
productivity, for example, might be the ratio of degree recipients to the 
number of program enrol lees. To complicate matters even further, most pro- 
gram evaluations, and we suspect the case will be similar for program review 
activities, employ criteria that require multiple indicators. Most program 
evaluations have severe technical problems when it comes to setting standards,, 
and we expect that PR will face the same dilefmmas whenever reviewers are 
asked, "How much is enough?" 

Lastly, to these three concepts, criteria, standards, and indicators, 
we would couple credibility. Unless a program evaluation is believed by 
tKose who must respond to it, there is little likelihood of utilization. 
And, the degree of utilization is one indicator of whether an evaluation 
is successful. , We believe thar the same case will hold for program review 
acti vi ties . - 

Moving from the realm -of evaluation per se, directly into. the gut of. the 
matter of program review, key criteria will require opnrational definition, 
since they will function as the dec ision nexus in all review activities 
whether these nccur at the institutional or statewide level. These criteria 
include: duplication, productivity, consist-ency, and e^iciency. In the 
act of definition, program reviewers will discover that these characteristics 
are not independent of each other, but quite to the contrary, they will appear 
intercorrelated in observations of program practice and effect. Thus, to 
believe that judgments on specific programs will be able to be made in 
fashions that deal with each of the criteria individually would be fool-hardy. 
T h e d e p e njle n,t_ criteria of progra m reviewwill have to be seen as intercorr elated 
di mens ions Of program excellence . 

The relationship between the criteria, standards, and indicators of a 
pro<)rdi!i (such as duplication and consistency .lat will be the judgmental 
array of the review process will ra.ise quest. .jns such as the following: 

[11 What criteria of postsecondary educationa-1 programs will be used to' 
, judge the fjrograins as worthy? 
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[2] Will these criteria be the same for programs at all levels of 
the educational structure-? 

[3] What levels of performance will separate duplicated from non- 
duplicated programs? » 

[4] What, levels of performance will separate productive from non- 
productive programs? 

[5] What levels of performance will s«parat^3 the consistent from 
nonconsistent programs? 

[6] What' levels of performance wiT! separate the efficient fr(3m the 
inefficient programs? ^ • 

[7] In general, What inputs, activities, outputs, resources, and 
ratios will indicate nondupl icative, productive, consistent, and 
efficient programs? \ 

Questions of this order will need to be investigated as part of the 
program review process or that effort will be doomed from its inception 
to a future of turmoil, political affrontery, and methodological imeptitude. 

The third aspect of a realistic program review effort will entail 
the distinction evaluators made between clients, audiences, and sponsors. 
In this instance, when we SjJ'eak of the client, we mean the person or group 
of personvfor whom the program review process is intended to produce 
userjj]_,><fTormation. By audiences we mean those' groups other than the 
cli^^^who have a stake in the program and w(io wish to share in the infor- 
mation that is 'generated because they believe the ijiformation and its 
possible effects on the program under review may have direct impact on 
their days. By a sponsor, we mean that' person or group whi. carries th,e 
financial burden of supporting the costs of program review. Careful 
distinctions between client and. audiences need to be made at the outset 
of program evaUiations, and we have no doubt that similar distinctions 
will heed to front the program review effort. 

The consideration of clientship in evaluation and in program r^feview 
focuses around the answer to the question, "Who is to decide?" This 
question becc:^ies an important role separator, but more importantly, the 
locus of aecision making becomes crucial when clients 'and audiences disagree 
about the decision. What happens, for example, when the local institutions 
and the state disagree about a program oriented decision? What will the 
decision rules be? What procedures for appeal ," re-review, and di/?pr 
will carry sway? Specifically, what will be done with programs that are 

wni they be given support for improvement? Eliminated 
will be done with programs that are judged efficient and 
they be expanded? Maintained at their present level of 
will the decision rules be? 



judged deficient? 
aUogpther? What 
productive' Will 
excel lence? What 



Consider the problem of decision rules more fully. There are multiple 
dimensions to program excellence, and there will be multiple decision rules, 
Consider, for excLuH^le, tfie criteria of consistency, productivity, and 
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duf).! ication. If we nest productivity under duplication and dichotomize 
each criterion, the decision grid suggested in Figure 1 is created. 

Figure 3 

Decision Categories Employing^ Three 
Dichotomous Criteria for Program Review 
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Examination of the grid in'- Figure 3 suggests that in two out of eight cells 
there should be relatively little difficulty making a decision, beyond those 
complexities that describe political reality. Cells three and six, for example, 
should both signal clear directions for decision. We will want to maintain 
programs that are consTstent, productive, and nonduplicative (3), and we will 
want to terminate programs that are inconsistent, duplicative, and underproduc- 
tive (6). Or will we? Will there be other more important criteria? If w^ 
enter a further criterion called program °qual ity, and if our program curr^tly 
sitting in cell six showed extremely well on our program quality indicator, 
although the program was inconsistently excellent, duplicative, and under- 
productive, would we be equally quick to consider requiring its removal? 

The Quandary. of Multiple Decision Rules 

To this point we have assumed that all criteria should have equal weight; 
that is, they should all be given equal importance in any decision. It would 
probably be more reasonable to consider weighting the criteria according to 
some preference function that will allow us to differentially maximize various 
outcomes. Consider what happens to our decision grid if we assign quality 
points to each of the three criteria according to some perceived relative 
importance, such^that the sum of the<iuality points equals 100. For example, 
suppose that a staters budget analysts rank the criteria in order of importance: 
1st, duplication; ?nd, productivity; 3rd consistency. And further, suppose 
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that these analysts feel that duplication is'more critical than both the 
other criteria combined, and productivity is about twice as important as 
consistency. By aj locating points according to this preference ranking, 
the following decisions would be made: 



Dupl ication = 51 pts. 

Productivity = 35 pts. 
LET • THEN, 

Cons istency = 14 pts . 



TOTAL = 100 pts, 



Change or 
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Drop 


Keep 
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A simSsUr pattern of decision would emerge in every instance where any 
single OviteKonNwas weighted more heavily than the ufhff^two combined! 



However, if tTTer-ftiree criteria were judged to be equally relevant, say 
by. a particular institution, then some decisions would be reversed as in the 
following system: 



TOTAL - 100 pts. 
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Accorling to the weightings given the various criteria, 'decisions about 
programs in cells (1) and {d) have been reversed. Moreover, the discrimina- 
tion among programs has di sappeareo , since all cells except 1 and 6 are now 
scored evenly. A critical point to recognize is that, depending on the 
woiyhtinqs of the criteria, any of'the six cells which are hot unanimously 
excellent or deficient could he maintained or abandoned. Program review will 
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need to determine how the weights will be set and by whom. Program 
review, will have to determine whether the same set of weights should 
be applied to the criteria across all levels of program review and - 
across all institutional types. State' level personnel, for example, 
may argue for different weighting systems than will institutional 
representatives. ^ 

De aling wi th Disagreement and Conflicting Values 

Once the problem of weighting of criteria has been dealj; with, . ' 
another important set of decisions needs to be^made. These decisions 
concern v/ays of dea^ling with disagreements or differences of opinion 
about the adequacy of program structures or even- the reasonabil ity of' 
a decision to continue or terminate a prograiii. Consider, for example, 
a two party comparison in which institutional program representatives 
are compared to state level program review representatives. The possible 
dimensions of agreement as suggested in Figure 4 represent the possibilities 
for one type"^ of conflict that program review activities may have to confront. 



1 



Figure 4 



The Dimensions of Agreement Between 
Institutional and Program Review Representatives 
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Evidently, in cells (1) and (4) there is little difficulty since both 
qroups hold the same opinion on the program, although we should point out 
that the reasons that they hold this opinion (what we called earlier, the 
practical argument) may be substantially different. Be that as it n^ay, they 
agree, theoretically, in cells one and four. However, in the off diagonal 
cells (3 and 2) there is opportunity for conflict and discussion. Decision 
ruins need to be constructed i:o control the numbers of programs classified 
in two cells [2 and 3) since without these rules, program review activities 
risk beconiing unreal istical ly expensive. If, for example, the state wishes to 
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certify and continue programs that the local institution may wish to abandon^ 
modify, or drop, it becomes harder for institutions to control their own 
resources and preserve what they may rightly perceive as a sense of insti- 
tutional autonomy. On the other hand-, if the state attempts to drop programs 
which the local institutioi^ wants, the state, may be in for a considerable 
professional entanglement. 

If there were a basis from which we could set some prior estimates 
(''priors") on the probabilities that programs will fall within one of the 
four cells defined in the agreement matrix, we might be able to come a little 
closer to estimating the costs of statewide program review in postsecondary 
education. Consider the following example. 

In the State of New York Regents Review of 126 doctoral programs in lO 
disciplines, 70 (56 percent) were assessed as high quality programs , 29 (23 
percent) as potentially high quality programs, and 27 (21 percent) were 
closed down or voluntarily withdrawn by the institution (Harrison, 1977). 
Using these probability estimates of the percentage of programs that will 
fit each of our cells in the little^2 by 2 table, we begin by* estimating the 
percentage of programs that are likely to fal> in the main diagonal cells 
(1 and 4). Let's assume further that the number of program decisions that 
fall in these cells are the number of programs that indicate' an effective 
and efficient program review effort. This is true due to the fact that as ' 
the frequency of programs in the off -'diagonal cells increases, the costs of 
program review, both of actual assessment and of political conflict, are 
predictably going to escalate rapidly. Ideally, place 95 (76 percent) of 
the program decisions in cell (1) and 10 (8 percent)^ in cell (4). Let us 
assume that the other two cells split differentially, such that there are 
about 4 times as many programs that the state wants to drof and' the institu-. 
tion waints to keep as the other wa^t around. We, then, develop the following 
estimator: 



State_J.evel 
Program Revi ew 
Representatj ves^ 



I ist i tutional Representati ves 
(126 Program!!^ 



Keep 



Change or 
Drop 



Keep 



Change or 
Drop 



17 (13:.) 10 (8%) 



Total 



Total 



95 (76%) J (3%) 99 (79%) 



27 (21%) 



112 (89%) • 14 (11%) 126 (100%) 



This outcome woul'/ mean that in the results of the New York doctoral program 
rt'vi<'W cycle alluded to previously, 17 (61 percent) of the 27 programs that 
were directed to close would be the object of further review, discussion, 
dr-qumnnt, and data collection. In addition, another 3 percent or 3 to 4 out 
of-' ')9 of the programs labelled "high quality" or "potentially high quality" 
would a 1 s .0 be a source of disagreement. The establishment of procedures for 
statewide program review will gain from efforts developed to estimate the 
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minimal tolerances for decision agreement and program decision accuracy that 
are necessary for the system of program review to operate efficiently. 
Incidential ly , the better a state system is, (i.e., the lower the total in 
cells 2, 3, 4), the less cost efficient PR will be. This is particularly 
true if every pror,ram must undergo detailed scrutiny and few programs are 
eliminated or made more efficient. 

Just i ce, Certainty, and the Value of Information 

o 

Three important questions for policy formulation and the development 
of program review procedures are now suggested based on an examination of 
this agreement matrix. The quest'ons are: * 

(1) Are the tradeoffs or conflicts that result (the off diagonal cells) 
worth the efficiency gained in eliminating mutually acknowledge^((k)or programs 

(cell 4)? 

(2) How much will it cost to get information which wil'' produce fair and 
accurate judgments and decisions? For example, decisions to eliminate programs 
are much weightier than decisions to modify and improve programs/ Thus, they 
need stronger evidence in order to minimize the probability of either a false 
positive (keeping a poor program) or a false negative (eliminating a good pro- 
gram) decision outcome. - 

(3) Strongef evidence means better inf ofmatl|)n, both in quality and amount 
(multiple measures across multiple traits). Is the value of the decision to be 
made worth the cost of the information required in order to make that decision 
accurately and fairly? 

In reality, the activity of statewide program review will likely involve 
not three but many criteria, and not two but multiple decision groups. Social 
real ities are complex realities, and, academic programs are social realities . 
People live there.^ In order to establish a just process for review, policies 
might usefully be established that protect the system against the worst types 
of lecision errors that could occur. Described above, the errors most to be 
avoided, are the inaccurate estimation of both excellent and poor programs. 
In the former case we risk harming an otherwise excellent program, and in the 
latter case we risk permitting an . ineffective and resource draining program 
to continue to waste people's time and money. One way co guard against these 
outcomes is to load the decision boundaries so that the probability of these 
tv/o errors is brought to a minimum. 

For example, in the certification of conimerical jet aircraft pilots, 
there are two decision errors that c jld lead to public tragedy or personal 
insult. One would be the appointment of an incompetent pilot and the other 
would be failure to appoint a competent pilot. In order to protect ^the public 
interest, confidence intervals can be constructed arouad decision bases so 
that the probability of making the first error, appointing an incompetent 
pilot, is 1/1,000 and the probability of making the second. type of error is 
10/1,000. The decision is loaded to protect the puMic go6d. 

Such probabilistic safe quards against specific decision errors v/fll provide 
one protection for program reviewers against the fallible evidence they will 
have to interpret. Beliefs in certainty passed long ago for most people 
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involved with social sciences, and the same is true for most program 
evaluators. We hope that program reviewers will approach their evidence^ 
with as much mistrust as do experienced program evaluators. To do anything 
else would be to risk turning program review into a rude empiricism and that 
^ '\ ^ would prove to be not only unprofessional but also methodologically inept. 

Some Alternatives ^ ' 

Evaluations typically raise more questions than^-they answer, and lest 
we be convicted of similar effects in this paper, the following suggestions 
are offered as warninga about some predictable threats. 

y , / ' 

Operating from a posture of power," mandate, and public responsibility, 
statewide program review efforts may be justified, as evaluations directed 
at insuri'ng external accountability within „and across postsecondary programs 
in education. PR is a- process designed to monitor*, regulate, and judge worth. 
How these outcomes will be best achieved is a consid^ation in design, method, 
and tactics. It is to that set of concerns that we now turn in pursuit of 
dsefiH things' to do in or^er to implement program revitw in the best fashion 
possible. 

Two Stages of PR: Old Data and the Marketplace 

It is probably a tr|6ism tliat available data is a lot cheaper than new 
data. Thus it follows tjhat anything program, review can 'do to maximize its 
utilization of on-hancU^vidence will be a valuable cost-saving technique. 
For„example, it may be that data already available at the state office level 
•and from organisations such as NCHEMS and HEGIS may provide the basis for a 
primary level review that will permit programs to be sorted into several cate- * 
gories that will discriminate'among programs according to their-, need; for 
further study, A procedure^ not unlike this has already been articulated for 
the^accreditation of colleges of teacher .education by David Krathwohl (1978).^^ 
As a result of a closer analysi s of available data^ "additional evidence of 
either a self-reported or survey nature may only need to be collected from 
those programs that appear to require more study or where the reliability 
and validity of the on-hand data were subject to sferious question. The 
second stage of PR would tiien be only for programs that were^judged to 
stand in Jeopardy at the end of Stage I. Actual site data and complete , 
institutional self-studies would be required of such pHograms. Programs 
which were judged acceptable at the conclusion of^a .Stage I review would 
generally be left to the vicissitudes of the marketplace. This might be 
especially appropriate where two otherwis^acceptable programs seem to be 
providing duplicate services. The market mechanism woul^ encourage such 
programs to upgrade and diversify their offerings in an attempt to differentiate 
their programs from each other. .Thus, the market could provide much the same 
stimulus to improve and/or winnow programs , without the cost of detailed PR. 

The Value of Information and the N ature of Evidence 

The spectrum of data p ssi^ilities is wide and spans the entire range 
of information gathering tecinioues described earlier. Tactics that range 
in scope from the traditional models of accreditation procedures (self-study 
followed by external expert judgmeats in a site visit model and \ sequence 
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of panel reviews) to the arguments from budget studies and audits all 
present pos si bile evidence to upport the questions that will focus the 
program reviev^ effort. The guiding concern in all decisions about the 
collection of evidence should be the .extent to which the costs incurred^ 
are justified in terms of the additional confidence or power such new data 
y^ill add to a de .ision. Robert Stake's review of measures and techniques 
for estimating objectives, priorities, and other judgment data is a useful 
resource here (Stake, 1970). . « 

There are also la number of vulnerabilities that confront program 
eva,luators, and they are- on the horizon for program reviewers. Some of 
the snost obvious ones include the presence of multiple audiences and their 
potential impact on any program review proceeding. Insofar as program 
reviewers attempt to develop systems that will be responsive to the varying 
perspectives held by differing audiences, the costs of review are going to 
.escalate. On '.he other hand, if these alternative perspectives on 'the worth 
of programs ere not considered, program review risks ignoring blocs of people 
v/ho possess considerable personal and political power. As the. effort of 
i^fKi<vidual i'zed methods to meet the specific if not peculiar requirements of . 
individual programs increases, the probability of there being a sta,ndard set 
of methods decreases. It may be because of considerations similar ito this 
that the- tradi^tional portrait of program and institutional review processes 
represents an accreditation-1 ike activity that is usual ly^ devoid of instruc- 
tioral outcome data and typified by the absence of stated standards and a 
heavy reliance on expert professional judgment and consensus. 

Inexperienced program evaluators frequently have aspirations ♦•hat are 
bigger than their budgets. Most clients have more questions than their 
budgets have monies to support the labyrinthine studies that are necessary 
to deal with the questions even at a descriptive level. A parallel case 
will exist for program reviewers if careful consideration is not given at 
the front end to the magnitude of costs involved in a system that will review 
all extant pos tsecondary programs within a state over a prespecified per,iod 
of years. Given' required visits. and follow-ups, the cost;^ entailed in such 
an undertaking in terms of staff time and budgetary commitment wiT be large. 
Thought _shc)Lri d_ be gi ven, c nseq uently, to whethe r all program''^ nee d to be 
'"^yjwe'li and whether there migh t be a wa y to consider voluntary review with 
^ art e r_ i n s tj t'utions as a means for pilot tes ting r eview proc edures 
PrjP/_t_o statfwicTe use. 

" Program evaluators with any experience will warn the neophyte that one of 
the irksome and time-consuming aspects of the practice deaT^with data manage- 
ment. The'skills and person time required to set up and maintain usable and 
efficient processes for da^l retrieval and anjlysis are considerable, and 'they 
cire frequently overlooked and underestimated. The enormity of the data base 
that will be rapidly developed in any statewide r^'view process ^.hould warn 
proqram rev iew ^^J i rectors ahead of time about the significance of the data 
mandgenent prol/lf_^ms that jwait them. The management of thi.s information, which 
i^j 'apt to develop wide appeal to individuals and agencies who f^d nothing to do 
with the ( ol lection (;f the information, will be one additional factor that could 
leal to an iricred':>e in, a spate's bureaucratic structure. In this case it will 

a structure tor pro^Jram review, one that will house a goodly numt^er of person 
primarily responsible to estabHsh, maintain, and u[)date i nf onna t i •h\ inveritories 



71 



\ 

\ 

If the (jdrcil lei s to prTjfjrani evaluation v/ork out as expectec', program 
f-virwcrs c-n 1 1 also tace the task of meshifsg multiple indicators of co. - 
rMliited c^rtorid in order to inform a weighted judgment of program worth. I 
ih'is i'i ru) simple task. If done ineffectively or incredibly, program review ' 
will doubt! experience the costs of failure in addition to the cost of 
1 'inclement If]'.) the d roc ess. 

The co'U . such failure may be observed in areas only remotely related 
to the dp.f)drentiy risk f r '.^e process of collectinx^ information. The politics 
ot program review and the possible effects it may have (from the alienation 
of college fcKulti.-^, to the weakening of educational over-sight functions) 
sifindl that people responsible for the d3sign and articulation of program 
review icttvities are gointj to have to be people who are intimately acquainted 
with the power and the politics of postsecondary education in this country,' 
particularly in their own ''tates. 

Pro^jrar' review presents a series of problems that are actually design 
Lonsider^ations . Program evaluators spend a lot of time developing alterna- 
tive ariuirents Vo deal with a^ unflinching reality called "what's '^ut there/' 
and It IS reasonable to expect that program review directors shoulu spend time 
in ^ii.'iilar arqunients. Research designs or evaluation designs, call them what 
/^ju will, are best understood a' a set of ■ careful ly calculated tradeoff s and 
aiiproprMte jiis^ i f ica t ioris that are offered in defense of design decisions, 
f ive of ^hr h lu-off". that may confront program, review are suggested in the 
tollowin-; tancifiil polarities: 

Tradeoffs Confronting Program Review 



i . f 1* , ^ Bandwidth- (, 

'•'rnqir.ji r Mw ier;Lv Cost of Reviews 

'> ■ .1* 1 v»' i: I dr i '..on • -- Criterion Comparison 

: ,prla^p^ — Quantitative Emphases 

> ' - - External Role 

ir^ tif ifil <l It tiutoriif^'., but w present theni in that fashion for the 

, \ r ^. ' J rin ! |:,Kif1vn rJth . The difference between fidelity and band- 
* ' »v^' *') ' <' .'.iKie 'If^M'jn and couynuniv:at ions engineering. Kecently, 
' M* v-'» ( riHiti-n h iT)/!), has warncl program evaluators of the 

. t ; Mill I'M' viif/ inr; .;pf.ifirjty (fidelity) anc general izabil ity 

• r^M'iurp insiqn. As we attempt to increase the clarity 

V * I wf^ di iinish the range or bandwidth over which that 

! ,^■t ; if : '.Mftrii f)r 'l>'tpit(Mt, Tor th^ <K.tivity of program review, 

n* 1 1 1 fpf cfK r !etw(»pf> bandwifttti and fidelity symbolize the dif- 
.! t rn"i,>! mfj t , fli'velop iiiqlily ^.pecific, iricroscopic determinations 
j -N'i'tijrr ( i ty ) ,r» ^Jifn()ared to broad, descriptive ■ 
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statements of proqran^ attainment (bandwidth) that alilow the categorization 
of programs. As an exciting departure frpm the conventional wisdom of the 
accreditation model, program review directors should consider designs, that 
permit both elements of high fidelity and others that maintain some bandwi^dth. 
Soi e postsecondary programs that have achieved distinguished records as 
successes or failures might be reviewed in depth in order to increase the 
firlelity of the Claims that can be made about the program and to determine 
w^ it makes them to be so. 

(2) Marginal Efficiency and the Cost^of Review s. The relationsh^ip 
between margi naf ef f i ciencies and the cost of program review varies inversely 
according to the ;^umber of audiences served, the number of general and 
specific criteria employed, and the planning horizon used. That is, the 
gain in efficiency from eliminating one additional program will be minimized, 
perhaps even null ified, if PR addresses many audiences, uses multiple criteria, 
and plans only for the short term'. This inverse relationship holds for the 
first two factors, many audiences and multiple criteria, because different 
airdiences need different data and it simply costs more t"' col lect- additional 
data aboutawide range of concerns. (This' point relates to the previous 
discussion of fidelity and bandwidth, where a study which holds fidelity 

onstant and increases bandwidth costs more.) 

Establishing an appropriate planning horizon, however, involves what is 
now recognized in systems analysis as a classic tradeoff between efficiency 
and effectiveness. Thus, long range planning, say a pi anning /horizon of 20 
years, might dictate seal i ng-down , modi fication , or even increases in support 
of some programs which might be more ef f iciently ,cl osed in the short run. 
It is fundamental, then, for program review to be ye^y clear/ about the costs 
of gathering data, the probability that major savings will be achieved, and 
the tradeoffs of short-term efficiency for total systems effectiveness. 

(3) Normative Comparison and Criterion C omparison . According to the 
central limit theorem, all frequency distributions tend ^o approach the 
normal as the number of mean unit samples increases. Mo$t program evalua- 
tions, and likely most program reviews, do not on will rj'ot possess enough 
units to capitalize on the known characteristics of the/normal distribution. 
One problem with this type of argument is that we never really know the 
distributional characteristics of variables. Thus, we tend to assume that 
thf^y are normal, when in fact they might better deserve a skewed shape. In 
fart, it is probably a credit to our postsecondary institutions that we can 
t)e co-fortable with the a^:sumption that the distribution of program merit 

is projat)ly negatively ske//ed as suggested by the Mew York doctoral review 
data and in Figure B on the next page. 

It thp d istr it)ution of program quality is skewed as suggested in 
F i^juro B, then approximately ^1 percent of the programs would be rated as 
h)w. ()n the other hand, if we begin by assuming that the distribution is 
standard normal, as depicted in Figure 6 on page 74. then the percent of 
prfjf|rains that should be classified as Low quality under distribution (A) 
would hp approximately 17 percent. 

ilnfortunately , what occurs in either situation, under this model, is 
that if wp lop off the tail of thf distribution that contains the low 
:|"Mlif/ i-TO'irams ir/ [)ultin'] thf}sr> ' programs oiit of business, what prui]rdr[} 
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Figure 5 

yy- Hypothetical Skewed Distribution of Program Quality 
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Figure 6 

Hypothetical Standard Normal. 
Distribution of Program Quality 
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reviewers will have to be wary of is that Lhe distribution will tend to . • 
reassume the same shape over repeated administrations of the program 
review effort (distribution B). This is not only a characteristic of 
the model; this is a primary characteristic of what has come to be called 
the norm-referenced model of testing and assessment. 

Problems occur when we start using the distributional characteristics 
of .thenonnal curve, such as the standard deviation, as the basis for 
deterniijiing the location of programs that should be dropped. As is evident 
from the differences between distributions A and B in Figure 4, cutting pro- 
grams that fell in the left hand tail of the B distribution would mean the 
eradication of programs that actually were of moderate to good quality. 

One way to circumvent this dilemma is to install a prestated, external 
standard of quality which is not free to vary and then to distribute programs 
around that standard. This is the kernel concept in criterion-referenced 
testing. The problem with this approach is that it is extremely expensi ve 
to locate the standard on empirical grounds. Thus, most standards, minim al 
<^0!HPjAgAcy_g_utt:Jng sco res, and the classification systems that the activit y 
of progra m review w il l devel op for loc ating acceptable and unacceptable pro'- 
gram perfor mance a re not empirically derived but judgmen t al. Such judg ments . 
are preeminently soci al a rtifacts . As such, they ought to vary over time 
with the changing premises of the social system that has created them and be 
responsive to the range of premises within the system at any given time. 
Onf? of the delights and one of the technical perils of program review may 
be that it will use human judgment to gauge some programs. The stretch in 
this human ruler or standard hopefully will not be the result of measurement 
error so much as it will be a response 'to true change in social expectation 
and belief about the .nature and presence of excellence. The ultimate pr e mises 

0 PrPlfJTL review wil 1 have to sta n d or fall on the cre d i bllity of' human judg- 
"'^ALjj Aui^PP.'^ted .in the opini ons of expert an d trus ted" o thers . The 'stretch 
of th(]se expert opinions will be shown by the extent tc which they are respon- 
sive to contextual as well as pervasive social differences. 

(4) Qua) ltd tjye^ Emphasis and Quantitativ e Emphas is. There is little to 
hv said about the differen'-e between qualitative and quantitative comparisions, 
bf cause the truth of the matter is that all quantities are qualities (Kaplan, 

1 Jb4). The cnti'-e debate that currently seems to have fractured the field .of 
e-ldcUionjI evdluation into warring caii'ps/ of competing ideologues is little 

« iiiijrp than ri difference of rhetoric and techniciue. The>'e is no sij*.K thing as 
"h,ir;l" fJcU.i .in/ r .re than thGre is "soft" data. All we have to go cn is 

rira 1 evi'lencr, and the acceptability of that evidence varies m^st widely ' 
Jt'lienM i ncj ij[)(in whom you ask or to whom you wish to be credible (cf. Mouse, 
1'*/;). f'r()(|rdi;' re-^iewers should sev.k credible evidence.- 



('.) Intfn-n,il Rf) 1 e druJ [ / terna 1 Role. Mi r iiael Sc r i von ( 1 967 ) dHtinf/uished 
iM'fwfcr' t.hr- internal ,ind extern. il roles of the educational evtfluator Forma- 
''vt' (ir hel[iin(j evdlii.itions are most oTten intern. il evaluations in ";hat they 
,irT j.^-r'f orruMl |iy [)(>f)()le v;hn dfp .u:tively involved in the day-to-day operate.;.! 
I*'vt'lnpf;icnf of t lie ()rnr]r,.-\i!i or product. As a result of this, tbv.ir credibility 
nfrcr, oiif'.idc the [irofjr.im is sometinies siis()e;;t. Thr y rre(|uefitl> will h^tve 
*■> ("'frn.il evaluatiori vjh'M sui-ima t i ve judqinpnts .ire to t'e i;ia e r,n t lie 

"'^'•'iM r''ii'i'- I! of thn [irfwjr.ii" , esi)n(i,illy one wlnr'n t.(V) h.ive spent 
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some paryof their lifetime developing. External evaluators, while they 
frequGf>^ly lack the bias that comes from extensive personal inves^-ment i.i 
a pp^rm, also risk the possibility of not being abl£ to come to Know a 
^^^gram int.iniately enough to support a fair, accurate., and credible 
judgment of its worth. As portrayed in the accreditation model, the 
processes of prograir reviewwill utilize botn internal and external evalua- 
tion. Tne internal evaluation will be represented by se/l f-studies and 
reports of previous institutionally sponsored evidence. \The external 
evaluation will be seen 1n the use of independent teams of outsi !e experts, 
reviewers, and panels. This process seeks the congruence and discrepancy 
between the findings of the ..sel f-study and the informed opinions of outside 
experts. 

Thus, as the activities of program review are designed, attention needs 
to be given to the tradeoffs between fidelity and bandwidth, efficiency 
and cost, normative and criterion referenced comparisons, quantitative and 
qualitative techniques, and internal versus external roles. 



Is Worth-find ing Worthwhile?: The Bot to m-line on Program Review 

It was appropriately asked of Plato, "Who shall guard the guardians?*' 
And it is equally well to ask^'of program reviewers, "Who shall review the 
reviewers?" The question is typically raised in program evaluation spheres 
under the term, meta-eval uat ion . Its. answer requires a description and 
definition of the criteria that ought to be used to judge the acceptability 
of a program review policy and process. In educational evaluation some good 
thinking has been completed on this matter by Daniel Stuffelbeam (N.D.), 
Michael Scriven (1969), and Lorrie Shepard (1977). Currently, a task force 
under the direction of Stufflebeam, based at Western Michigan University, is 
nearing completion of a set of standards for educational evaluation. These 
standards will give useful suggestions to program reviewers when they begin 
thinking about the meta-cri teria t^hat shogld dominate their processes. With- 
out justification, we offer that t^e activity of program review will have to 
be acciDuntable to questions about completeness, fairness, accuracy, and 
u t i 1 i ty . 

Based on arguments authorized by Ernest House* (1978) we would suggest 
that the utility of program review, allowing argument from authority, will 
coniG-pol i tica"* ly to rest on persuasiveness. Program review will succeed in 
those contexts where there has been an effective communication of 'beliefs 
and value dispositions about the programs under study. This will mean 
Liirefully planned and conducted reviews that will require state and insti- 
tutional personnel to expose, discuss, and defend the first principles of 
thpir programs and review processes. Failing this, bcth House's analysis 
and uurs predict a failure in utilization, and consequently, a failure of 
pTtu. t.'i^a 1 ar()uinen t . ' 

Thi^ analysis has id(^ntified a number of issues that are related to 
pro(iran rovi^'^w policies and processes. They can t)e effectively summarized- 
<is t]ups t ion'". . 

(I) »ifiat cr i t'Mia , standards, and indicators will ho employed? 
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(2) Will the same criteria and the same standards be used to assess 
programs across universities, colleges, community colleges, proprieta'^y 
programs, etc.? 

(3) How will duplication, efficiency, productivity, and program quality 
be operationally defined? 

(4) How will efficiency be related to effectiveness? For e. ample, if 
programs are eliminated now, what is the possibility that they will 
have to be reinstated at some future time at much greater costs? 

(5) What prob<ibil ity of error will statewide program reviews be willing 
to tolerate with respect to the possible elimination of a good program? 
Maintenance of a poor program? 

(6) What will be the costs to the state required by central planning 
and the establ ishment of structures to support the massive data flow 
that statewide program review will create? 

(7) How will program review handle the aggregation and analysis of data 
on multi pi e cri teria? 

(8) What are the important audiences that program review will have to 
address, and how are these audiences disposed toward the review policy 
and processes? 

(9) What are the possible unintended effects of program review? To 
what extent, for example, may program review lead to a homogenization of 
postsecondary edi cation? 

(10) What will program review say to faculty, powerful alumni groups, 
and others who believe that they have been injured by the closing of a 
program under state requirement? 

(11) What estimates have been developed for the costs of program review 
that will have to be covered by institutional budgets' 

(12) What procedures will be installed to insure comparability of 
processes across sites, programs, and iterations? 

(13) flow will the question of meta -eval uation be resolved for srtatewide 
program review efforts? To whom wi i . program review be accountable? 

As in the process of program evaluation, answers to these thirteen 
questions will have to be justified within the frameworks of practical 
artjumpnt.s . arguments which will inform the policies for program review, 
ihe.e anjunipnts will cofttain the beliefs and judgments of people. Hopefully, 
they will b" well-informed judgments and will constitute compelling arguments. 
Su(iqestion5 on how to cope wrth these and similar issues will probably surface 
from thp initial forrays of the states into the program review effort. Ptner 
v/iMluiii i-s ,.iv,ii lable from state level and school personnel who have already 
">4H»rif'nt 0(1 thp rpview process irthe K-12 sector, and these lources of policy 
d.Ud shoul(l not i)( overlooked. 
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Ld'^tly, dnd most importantly, if there is a final caveat to be taken 
from evaluation practice, it is captured in the public recognition of the 
central role that human judgment will play in all that has been suggested. 
Judg ment is uur refuge in the absence of certitude. If we could be certain 
about t he qu al ity of an ac ademic program, we would not need to rely on 
judgmen t. But it is our peculiarly human condition that forbids that and 
causes us eq ually to won der a t the frequent precision and reliability of 
those Judgment s. As most undergraduates will tell us if we ask, it doesn't 
take a Ph.D. to tell the difference between a good professor and a poor 
one, and we might add, between good and bad programs. The problem with 
judgments lies not in the naking of them. The problem with judgments and 
thus with proqram reviews is getting people to believe them. 




ERIC 



79 



REFERENCES 



Blackburn, R.T. and Lingenfelter, P.E. "Assessing Quality in Doctoral 
Programs: Criteria and Correlates of Excel 1 ence. " Report for the 
Regents Commission on Doctoral Education, Albany, New York, September 
1972. 

Cronbdch, L.J. Essentials of Psychological Testing . New York: Harper 
and Row, 1970. 

Cronbach,- .L.J. Designi ng Educational Evaluations Stanford University, 
(Manuscript in prep ^^ation, 1 978 ) . 

Dewey, J. "Theory of Valuation." In Th e Unified Encyclopedia of the Social 
Sciences . Chicag©: University of Chicago Press, 1939. 

Glass, G.V. The G rowth of Evaluation Methodology . American Educational 
Research Association Monograph Series, No. 7, Chicago: Rand McNally, 
1974. 

» • 

Guttentag, M. and E.L. Struening. Handbook of Evaluation Resecfrch . Vol. 2, 
Beverly Hills: Sage Publications, 1975. 

Harrison, O.G. The New York State Regents Doctoral Evaluation Project 
1973-77, October 1977 (mimeo). 

House, Th e Politics of Educational Innovation . Berkeley, California: 
McCutcha-n Publishing Corporation, 1974. \ 

Houso, E. The L ogic of -Evaluative Argument . Monograph No. 8. The Center 
for the Study of Evaluation, University of California, Los Angeles, 1978 

Kaplan, A. Th& Conduc t of I nquiry. San Francisco: Chandler, 1964. 

Krathwohl, D. "An Accreditation Proposal." In J ournal of Teacher Education , 
1973, 29, 28-32. 

PcKje, E.B. "Seeking a Measure of General Education Advancement: The 
Bentce." In Journal of Educational Measurement, 1972, 9, 33-43. 



Reyf-nts of the State of New York. "Meeting the Needs of Doctoral Education." 

Position Paper 19 (no date). 

Riker-, W.H. and Ordeshook, P.O. An Introduction to Positive Political Theory 

New Jersey: Pr. ntice-Hal 1 , 197 J 

Sc. riven, M.b. "The Methodo! ■jgy of Evaluation." In ACRA Monograph Series on 
• L valuation. Ctiicago: Rand McNajly, 1967, 39-83. 

'nriven, M.S. "An Introduction to Meta-Eval uation . " Educational Product 
Repor-t (February 1969), Vol. 2, No. 5, 36-38. 



80 



Scriven, M.S. "A Checkli^for the Evaluation of Products, Producers, and 
Proposals." In W. J. Popham (ed. ' ilu ation in Education . Berkeley, 
Californi-a; McCutchan, 1974. 



Shepard, L.A. "A Checklist for Evalua' 
Paper 9 in Occasional Paper Sen 
Education, Western Michigan Univu. 
April 1977. 



arge-Scale Assessment Programs." 
valuation Center, College of 
, Kalamazoo, Michigan 



Stake, R.E. "The Countenance of Educational Evaluation." In Teachers 
College Record , 1967, 68, 523-540. 

State, R.E. "Objectives, Priorities, and Other Judgment Data." Review of 
Educational Research , 1970, 40, 181-212. 

Stuff lebeam, D. "Fvaliiation as Enlightenment for Derision-Making." 
^ In W.M. Beatty (Ed*.), Improving Edticational Assessment and an Inventory 
of Measures of Affective Behavior . Washington, D.C: Association for 
Supervision and Curriculum Development, NEA, 1969. 

Stuff lebeam, D.L., W.J. Foley, W.J. Gepavt, E.G. Guba, R.L.VHammond, H.O. 
Merrinian, and .M. Provus. Educational Evaluation and Decision Making . 
Itasca, Illinois: F.E.u Peacock Publishers, Inc., 1971. ~i 

WiTdavsky, A. The Politi c s of the Budgetary Process . Boston. Little* Brown 
& Co. , 1974. \^ , 

Worthen, B.R. and J.R. Sanders (Eds.) Educational Ev a luation: Theory and 

Practice. Worthington, Ohio: Charles A. Jones Publishing Company, 1973 



1 



Relating Role and Mission to Program Review 



MissL^n, role J and scope planning and program review. 

J. Kent Carjthers 
Senior Associate 
NCHEMS 



83 



RELATING ROLE AND MISSION TO PROGRAM REVIEW 
J. Kent Caruthers 

I approach the topic of how institutional role and mission relate 
to the program review process from the perspective of mission, role, 
and scope planning. Over the' past two years, I have directed an NCHEMS 
project that examined mission, role, and scope development procedures 
used by both institutions and state postsecondary agencies. In that 
project, we have conducted ten case studies rf various institution and 
state agency practices in defining institutional mission, role, and 
scope. From these case study experiences, we will be developing a 
procedures handbook during the next several months^ 

I also view the relationship between role and mission and program 
review from the somewhat broader perspective of strategic planning. 
NCHEMS is just embarking on a four-year Strategic Planning program of 
which the Mission, Role, and Scope Procedures project will be the first 
major link. We consider strategic planning to be an .effort to identify 
viable matches between an i nsti tution ' s strengths ana capacity and the 
demands and constraints placed on it by the external environment. As 
such, the strategic planning concept provides an umbrella under which 
NCHEMS will study mission, role, and scope procedures; program review; 
community needs a>sessmen't; enrollment planning and management; insti- 
tutional envi ronment assessment ; academic outcomes; and a number of 
other related topics. Despite these many analytic, components, I con- 
sider strategic pi anni ng--and especially program review and role and 
mission plauning--to be primarily people-oriented processes. 

T heo r p t j caj^ Relati onship 

Llefore movirrg to the results of our case studies, I believe it is 
importarit that we first consider the potential relationship between 
program review and iristi tutional role and mission from a theoretical 
perspective. From this viewpoint, there should be a state of strong 
i Mterdependency ; each can, and should, build from and contribute to 
the other. In the best of worlds, this would be true for both insti- 
tutforis arid state agencies. When an institution's mission is well 
es>t:r)bl 1 shed and understood, it creates a frame of reference for assess- 
ing prograni quality. For example, a college with an "open door" mission 
would not be riearly as concerned with traditional measures of the quality 
of its enter-ifig students as would a selective admissions college* Or, 
.!M institution with an; urban focus may wish to place greater emphasis 
on tfip role of .^ppl i ed research and development than on basic research. 
This pinph.isis, of course, v/ould be in sharp contrast to that of most of 
n»p innrp t r^d i f. i onri 1 mnjor research universities. 

When ,Hi i nst i (-.ut ion ' s missiori is ambiguous and needs to be re- 
f^vi 1 1] 1 N».J ^ infornMt. ion--bnth objective mv\ subj ecti ve-- that cones from 
progr .ici ?'rvipw dctlvitips provides dn effective statement of what the 



institution actual/y is. For example, when an institutikon learns t^pugh 
prograri review U<at it is excellent in teaching* and only average in ^\ 
research, it may well choose to accentuate the pos'iti.ve. Likewide, an 
institution wbl^se program review results tell it that its science', 
programs comprise its real strengths may determine that assuming a 
polytechnic mission is the best strategy for it to carve out its 
niche in the system of. higher education. 

For the relationship between program review and institutional role 
and mission to i.iake a major difference, however, they must not only be 
strongly related to each other but also to the budgeting process. Only 
when these two processes individual ly and collectively can have influence 
on resource allocation is their full potential realized/ Ideally, any 
new resources available to an institution should go to /strengthen weak 
programs that have been determined to be essential to/che accomplishment 
of the institution's mission. Or, for those institutions in less fortu- . 
nate circumstances, butlg^et cuts should be made at the expense of those 
programs that have been'determined to be comparatively weak and non- 
essential to t^he institution's mission. / 

Actual Relationshi p at the Institutional Level 

Moving from' the theoretical to the real world, there is some evidence' 
that program review and role and scope information* have been used jointly 
at the institutional level in budgeting and priority setting. In preparing 
for its ten-year visit from a North Central Association Regional Accrediting 
team, Oklahoma State University measured its success in achivwinj^- its mission,, 
reviewed the strengths nf its many academic areas, and determined that a 
reallocation of financial resources was necessary (Robl, Karman and Boags, 
1976). This was done through a 1 puercent across the board retrenchment\and 
the selective strengthening of programs in those areas most essential to the 
/accomplishment of its mission. Austin Peay State Unirersity linked an 
informal needs assessment effort with program review to identify a viable 
mission for itself in the coming decade. Importantly, it followed up by 
internally allocating resources to fulfill its new role. And Chattanooga 
State Technical Community Col lege,' which has the typical commmunity college 
mission of serving its local area, uses needs assessment information to 
start up new programs and program review information to determine when 
programs can be terminated. They, too, are not reluctant to let financial 
resources follow program decisions. 

The link, among program review, role and mission, and budgeting is perhaps 
pven more important when an institution faces a financial breakdown. The 
Albany University Center of the State University of New York relied heavily 
on its own nternal p'^ogram review results and a newly develoDed understanding 
of its role and mission to identify those program areas that could he elimina- 
ted wh',^n it was faced with a state imposed requirement to reduce its 
budqpt by nearly 10 percent in a single fiscal year (Shirley and Volkwein, 
r)7.'0. Aquinas College, a four-year liberal arts institutions of approximately 
1,000 students in Grant Rapids, Michigan, chcse d somewhat different strategy 
(Hruby, D/l). After several rears of declining enrollment in the last 1960s, 
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it undertook a careful institution-wide program review to determine its 
strength and capacity. It found ways to change its mission to attract 
<iew students and to grow and prosper during a period when other institu- 
tions were falling by the wayside. 

Unfortunately, examples are at least equally, and perhaps more, 
numerous of where an effective link among program review, role and mission, 
and budgeting has not been achieved. An all too common example is the 
(Drogram review activity carred out by a graduate faci/lty council which 
uses qualitative standards that far exceed any realistic aspiration of 
the institution's mission and resources. As a result, it never has much 
impact Qn the budgetary process or the future of the institution. 

The reason for the failure to achieve a closer relationship at the 
institutional level among these three activities can be found through a 
closer examination of each. Most current statements of insti tutional 
mission, role, and scope are far too ambiguous to^^rovide a usefuJ^-Con- 
text for evaluating academic programs and establishing budget priorities. 
We hope the fruits of the NCHEMS project will be a first step' in r^esolv- 
■^ ing this situation. Also, the many conflicting demands that underlie most 
PCdgram review activities almost inevitably lead to less than useful • 
results. We are just beginning to develop an understanding of the com- 
plexities sur*rounding institutional program review. Approaches to budget- / 
ing in colleges'and universities also contribute to the sometimes distant 
relationship. This situation has become more critical in a period of 
increasingly inflexible resources. PPBS, Zero Base Budgeting and Per- 
formance Bucjgeting approaches have been developed in other organizational 
settings to respond to similar problems, but so far their transfer to 
higher education settings has usually been less than an unqualified 
success . 

Relati onship at the Sta te Leve^ 

Generally speaking, there is" even less of a link among program review, 
role and mission, and budgeting at the stat e level than at institutions. 
This is particularly true for the review of existing programs. While one 
can always hope for improvement, this failure to achieve an effective link 
may be at least partially explained by the limited role and authority of 
the state-level agency. 

A common form of program review at the state level is the so-called 
lateral process. When using the lateral review approach, the state 
T» aqency reviews all programs in the state in a particular discipline. 

The agency often invites a team of specialists from outside the state who 
are ticknowl edged leacljrs in the field to come and assess each program. It 
IS this particular program review design at the state level that appears 
tn prohibit an effective linking between program review results and insti- 
tutional rolp and iii-^sion The natural tendency for outside consultants 
who (irp looHng at seemingly comparable programs at a number of institu- 
Mons i'-, to t)nstr,i5t cne to the other. Although some lip service is usually 
given to consideHng how each individual program reinforces institutional 
inis'iion, the overwhelming tendency is to rank order the programs against 
.1 (onimon norm that is independent of their role in supporting the in^'.titu- 
t ion's basic purposes. 
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i -M'r *u sii.jijr,t, ^ w**v^f'» thai fhoso state agencies with 
P^'''-;'"'-^ ''rvHt-.v f -.(U)n':.ibi ] !^ ^,hoiild j^('ui:rjp their current approaches. 
Snw.i^ f'lf uj.ipdl rea :(;n» t,fi,i t many tif these agencies are in the pro- 
jrn' r>'viHv* ii!i >inf»'/. i'> t.u o I ici i ria to costly and unnecessary program 
•ijp i )• (jfi , -:ui) [i.u' iS(Mv> anionf) pruijraifis seepi in^-^vitablG. But the agency 
''''^'•^^ rrMiiifil jv.fif tf]tii for proqr duplication -to be unnecessary, the ^ 

..1^f'0t [i^* anit.ul tj) tho dchi*?vement of the i nsti tut ioh ' s ' 

^■'^^ t/i.r. of tMjilgptirv^.firacticc^^ usually followed at the state level 
>il'>'.. Ir.'if trr^j ..ltMjr>.v of linkajp u^ioncj prograi'i review, role and mission, 
iful I'l.d'jeMfh;. As conti^asted to i ir. t i tu t iona 1 budgeting practice, state 
lr'vi;l ji;^ !rvj i porforiiicd in hi<jhly aggregate teuiis and, in fact,! 
■^i;(j:jt unp-'M.tlf I'rw states u:.f^ some type of budgeting formula for dis- 
^ ^^'i: iti^>", nt resources to individual campuses. Given this budget- 

^^■1 ; , s"ire offkials h.wo few opportunities to make the budget 

' ' '^-^ ' ^ ''^ ^^'^^'^ ^or cwrrecting i nd i v idua 1 'program dpf iciencies, ' Even I 
t^iD^^'jh a 'i^MjttM' linkigc' iiMght be desired, its cost vjould surely be felt 
• .r'^Mtrf state level nkinagement of day-to-day institutional 

c ^ 

'"'^ — d 1 ^ t i(. lij L iVs ifi achieving a strong link among program 
^"^^ ^^'d fnl'.sion, cind budgeting at the state level, a couple of 
'''•^'*; ' ^ ^ ii^ind wher^e state Iov( 1 agcficies have achieved seme 
".iMn; these anaV/tic tasks mutually re-enforQi ng . 

[•^'■'t Mvf' fi) MX years, the staff of the Florida Board of 
^^'^ ; 'Ver'ftire; tiofl/ for the State University System, has set 
r.i l^- t aM per..rr)t(e)f^ of Us total bijd()Pt to be aMocated outside 
^^r:iil^ * a' '.[um j,H f.regramfiiat i c nee Is. The percentage set aside 
)!i •.■r:^-jaM tn.it )t doesn't destroy the basic equity of fund dis- 
a^hw-;''d by t.hf^ formula, yet the pool of protected resources 
■ e: »"r^)'i;N ffiaf prf igrsimma t i c change can be implemented. In particu- 
' 'J"' -rsivrrsUy System f-as established what it calls "programs 

'* ' institution was enc )uraged to propose, for special 

I' .--'"al i rograi:) ara^d-. whf^re it felt that it had a real oppor- 
' i MPvr fifUional (a'Mcj raaice . Their* proposals were reviewed by 
' 'es.uj;)' arul, fitter several years, at least one progran- 

•■ 1 ■ 1' ' ^ -.1. ^ r^wu I ,it Mr-i'ti n;'.t i tut ion . As the ins ti tution made 
''-iM' :^Mri\ tf) f Mr .'a!^' i'^.if^i, ajut as that body madn its selections, 
f ni . t i t ijt 1 (in.i 1 r' I ion ,if,d [>rogra!ii revii / information were ' 

^■••^ *.i * . in I'.atiM'i ,1 i.ij.i'a^i M,(j (jfn isiofi (UetweHer, 19/5). 

' ' ' ' ^M<i^»*r- itiori '.)rrMssif)n has fonducted'an interesting 
■ ' i^' ' M nr ::,:!;. f ! ■ , i- 1 j i t i f) ; i /.'Ver the [)ast sever/,1 years first with 
■a . : ' ^ rh'... w, ' .'.it(. rjr j!,ir\ ( [:o(juf^ and Troiitt, TT/H). In 

-'t ' rr,.irj, aii^iill .ii'ilar- tf) the Flori-da e^nerience in that 

' ! , ' 1 [Man, to a'.ide a ' 'Tia 1 1 [n'rr.entage of the 
1 , . ' ■ ; ' ■ ! * ^ ^ ^a * I. ^ fit 'M . t r i tjiit pd recording to formula. Also 
' • I' (' i/JiMjt 'W! wiM nav^ an o()()('r tani ty to recr-iv^' its share 

= 'S ^/rd', I'.* rithi^r' Miari ( r < .! t f profjf acis (.^f" d i s ' i rif t i on , the 

■ ■"■ i", 1 r, }, ,! < Kir d I na t i n<| t)od/, will ^iiloweaMi in'stitf 

' ' ■ ' ' ' * 1 I 'f ^ ) f ' I f I ^ 1( ^ 1 I ■. I f I ar ( '(I () f ) t \ own ( lioos ui' j , sub j »;( t 
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only to the requirement that the specially designated areas be important 
to the state-approved institutional mission. Then, when the institution 
can demonstrdt.e that it has met its performance objectives, the financial 
reward will be released. 

A much clearer link between program reviewand role and mission can 
b(? observed at the state level in the approval of n ew degree progr ams. 
In almost every state, the proposal d'ocunient which is filed prior to' be- 
bqinning a new degree program requires that the institution demonstrate 
how the new program fits its current mission. But even here there is 
•oom for some improvement ' since many state-approved statements of ins£i- 
' itional mission remain sufficiently ambiguous to allow an institution 
b justify ali'iost any new program. ' 

Towr '''^ Acjli_evj_,nc|__d_ strong er Link 

. ne key problem that blocks achieving a stronger link among budgeting, 
progra; review, and role and ni><sion is the relative underdevelopment of 
the sta'p-of-the-art in each/rea. Even the advances that have been made 
during- tM" [)ast decade may n^t prove especially useful during the coming 
period of .tabiiity or decline. 

Througn>iut most of, the hi'^tory of public financial support of higher 
edu(,,uion, colleges and universities have been in a period of growth--a 
fact that is reflected in the existing budgeting processes. Even in the 
nL(.a', ional year when financial growth did not occur, the situation was 
v'iewed a , a temporary inconvenience rather than as a signal for a basic 
roal i()nnient of t^? institution's mission. Also, most of the earlier 
periods of adju-.tment occurred 'when institutions were relatively small and 
if wa'-, possible to integrate the missit i and program plans through the 
t.uifiet without the sui^psrt of elaborate systems. The zero base budgeting 
./stf.T! has been promised as^ a panacea for budnet planners in this new, 
;"<jr.> Lodiplex, environment. But to date, there has been insufTicient 
t^-tiri'i uf /FJi'ir, higher education settings to determine its utility. 

tt.r at ii, pro'iraiii r'eview is a fairly recent phenomenon in higher 
'■du'. if ion, ^'Specially at the state level!. As a result, we are just 
''•"i i fn ■lo(.Linenl tlie different ways in which program review might 
h" . onjiii hMl, And an understanding of the best matches between the 
vi'-icn' pijrri.jsps for [)rograrii review and the processes available is only 
t"") i r:r; i n-j 1 1 , cnecgc . 

In . n ? ut 1 r'olf^ and mis'/ion planning is an n less developed 
"■'■a M i, Minfused t)y the fact that on the one ,id, institutions 

'.faf'/i-nrit s of ro]f> and mission since recording their original 
I'l'l 1 siiing their firs, college catalogs. But on the other 
■ P'lt'li' 1/ sufjported institutions of any significant size have 

i '■■.'<"! .1 ,••11' (• of missi'jn that is shared by all members of its com- 
'■""N' and !■. ey[)licit enough to serve as a guide for program review 
•"I'i !'ud'jit 1 n-i . Tfiis situation, of course, led to the creation of the 
'i' ^'lM'. Mission, Kolf, and S( ope f'ro( edunes project several years ago. 
■ ' ' iin.|!j( tifi'i tile (,(is.' studios and developing plans to write the 
'windl.iMil. , w" h.iv ( o I] o( t orl our i fioijfjfi t s in these five ar'eas: 
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• organizing for planning 

• types of participation 

• types of special analyses 



• comniunicd ting progress and results, and 

• maintaining a dynamic mission, role and scope identity 

My closing cominents will attempt to describe how current practice in 
each of these areas might be changed to achieve a stronger link with 
program review. 

Under the area orga nizing for»planning , we include such topics as 
determining an institution's readi^ness to plan, articulating the assump- 
tions that underlie the efrort, designing the particular planning process, 
\identifyinvj decision-making points, clarifying the objectives for the 
ac-tivity, and establishing a schedule. Aside from the general observa- 
tion that getting ready to plan is time well spent in achieving the desired 
outcome, we have seen that most major examinations of an i nsti tuJ!:ion ' s 
mission, role, and scope have occurred in response to some strong external 
stimulus. In most, cases, this has been the arrival of a .new chief execu- 
Mve officer. But in other cases, it has been the resutl of a legislative 
mandate, a severe financia'. . nsis, or preparation for a regional accrediting 
visit. The fact that most mission, role., and scope' planning is approached 
d ore-shot effort appears to create difficulties in achieving a strong 
link with program review. Whereas program review activities occur on a 
fairly cyclical basis (perhaps annually), attention is focused on role and 
scope only once every ffve or ten years. As the length of time since the 
last role and scope effort increases, l.he extent of agreement about the 
rnissiori diniriishes arid it becomes less valuable as a framework for program 
review. 

In the arcM cf types of participation in a mission, role, and scope 
deveJopment process, we have observed that the most successful experiences 
have relied on fairly wide r.cale involvement. > Participants typically ' 
in<Jude those who arr^internal to the institution, such as students, faculty, 
staff, administrators, and members of the bdara, and to a lesser extent, 
those who are external-. Types of participants in this latter category 
incluJe consultants, members of the geographic community in which the 
institution is located, and educators from.^nearby institutions. We believe 
such broad involv*?ment could also be important for achieving a strong link 
'otwppn M]lo and mission and program review. As an institution or agency 
(.i i .iihievp a cjrpater derjree uf involvement by the same individuals^ in both 
processes, the understanding of institutional mission should havc'i greater 
influence as a program review criterion. j 

Special ina lytj_cal_jtudies also. seem important for a succe^rsfyl 
mi ss ionV role, and scope planning effort. One such analysis th(^>^s 
bnen mentioned throughout this paper is aradc^n'c program review. As we 
mpntioned in the opening section, good program review information is 



89 



important in (issessing v^hat the role and scope of the i. stitution tu- 
ally is--as contrdsted to what its mission says it shouK be. Other 
types of analytical efforts which we have observed to successfully con- 
tribute to th^' mi ss i on , rol e, and scope planning effort are variations 
of cofinnunity needs assessment studies, cofnmunity impas^ studies, demo- 
graphic and enrollment projects, use of the Institutional ]&oals Inventory 
and campus environment scales, financial plans, and field trips to other 
similar institutions. Most of these analytical efforts might provide, 
important input not only to mis'^.ion, role, and scope planning efforts, «but 
also to program review processes if they are carefully planned and coor- 
dinated. 

Cormunicating a planned change in an institution'*^ mission is a 
conflict-ridden endeavor. Individuals are able to ccrr )are their own 
educatianal and social philosophies with those of othL^rs. They learn 
of discussions that concern their status within the orgrani^Lcition , and 
in ^.ome 'cases, ever: threaten their personal economic security. Probably 
no approach to communicating the results of.aiy serious mission, role, 
and scope planning deliberations can avoid such conflicts. But we have - 
obs(M^vpd that the unrest is minimized when there is open and freguent^ 
comniunication and the Ojportunity for a fair hearing of concerns. We 
have seen the effective use of special publications, public hearings 
in the cuniiiunity, and campuswide planning retreats in our Ct^se studit^s. 
We believe that this observation should also be of interest to those' 
who tf)ordinate program review activities. Further, we believe that the 
(.onfidence in both mission, cole and scope planning efforts and program 
review activities will be enhanced when members of the campus community 
t.ar^ st't' that they reinforce each other in shaping the campus budg-et. 

Mai D \a i rii a dyndmic mission, rolej and scope^ie nti t y for an insti- 
tuUor of hiqher education seems difficult to practice. While it is easy 
to ,it at one'., de^^k and draw f^ov^ charts showirvg how there can he'feed- 
t)ai t lou[r> from \norter term plan ing processes to the longer term mis- 
. itui , r'o h , and scope [)lanning effcji \ in practice this is difficult to 
a* hipvp. Ihf-rp ap[)pars to be no ea' y remedy to this situtation other 
^ru)n thf S];;:plp advice to try hard* /\ Areas of needed improvement ^ 
ir,v ]\v\p drVf^lof)ir)q more explicit statements of institutional mission, 
r-f' 1 p , arwj '.ropr ; preparinc] program review i^ecommendalions that suggest 
• or-f^ sp(^( iti( rof-iediat ion* efforts, and constantly asking oneself if 

•jr^ronf doMsiinis support or requir'e modifications to the ' i ns ti tut i on ' s 
i^-ission. I don't unaqine that many i:'^ t i tutions' wi 1 1 ever be totally- 
»^ttp(Mv*^ m ai I oiii()l i sh ] ng th^'se things. Deliberating on a college's 

i.sinn, r^M 1 (> , and su)[K,^ is t in^e-f.onsumi ng iMjsiness and time is becomino 
.HI 1 nt rM;>a s i ru) 1 / pr^p( ioijs commodity in this area of rapid change'. None- 
'"a I n ta i ri uig a rlf\irf^r sense of vision of wher^ the institution 
IS and should t;f» hfsiding is so impor^a tt that it deinarwds wfiatev^r time 
c^Mjijir'twi f rnfs senior^ tM.ad(-Miii( [^lacMf^rs. 

In V losing. I would likp tf) r^Mtfuate that I l^eliove pro(|ram review 
and "nssion, ro h • , and sc ope [)! arm i n<r can and shf)u]d j^f f ect i ve1 y sup[)ort 
p(«( ti Mthf'r\ Si institutions anrl (U|(uuins hav(^ alr(Mdy fnadf^ consider*- 
aMf [jro(|rTss in ttiis aroa. ! lu;po thn NCfU.MS mission, role, and si 
i^r-iu piiiiFMs handbook will b*' one tool that < fin h(^lp thoso'who (ir^(^ U*s 

/ 
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aJvanced learn of such prbgross. Also, through v/orkshops such this 
knowledge about the state-of-the-art in pro^mam review is rapid! 
spreading. This should help even the most a3v*rr6ed institutions land 
.agencies to keep learning and improving. I believe their renewed 
efforts and those of leaders in other institutions and agenciesJ 
toward achieving a closer, more practical^ relation between role, |and 
mission arid program review will be well rewarded* 
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STATE LEVEL PROGRAM REVIEW 
Wi 1 1 iam Chance 

Washington first becanie involved in reviewing existing programs in 
1971. In 1973 we conipleted the review of 135 graduate programs, and in 
1977 we completed the review of 360 graduate programs in our state. 
We've reduced the i nventory. i'n the public institutions from 450 by about 
93 or 94 programs. 

In the review of both new and existing programs, I believe that state 
level reviewers can play a complementary role with respect to institutional 
reviewers. "In general, at the statp iovel we focus on qualitative consider- 
ations--need for the program, program cost, and insti* Jtional role--but we 
cjet a great deal of quantitative information to back these up. 

I want to talk a bit about the four areas that represent the major 
areas of state level concern in my state. First is the quantitative/ 
qudliiative dimension. Our experience has shown that descriptive material 
on programs tends to be objective, or quantitative, in nature. Obiective 
information is comparatively easy to assenjble, while the subjective aspects 
>f a progran are much more difficult to convey. As a result of this we 
find ourselves dealing with what I call the quantitative/qualitative fallacy 
It operates something like this: Since the reviewing agency emphasizes the 
quantitative aspects because they are objective, it accordingly deemph^isize^ 
qualitative aspects because they are subjective. Programs are thus not 
evaluated on the basis of quality, b'jt on the basis of productivity, 
Therefore, a productive program of inferior quality will be continued 
while a high quality program with low productivity will be placed in 
jeopardy. The nwjor problem with this contention is that it separates 
(luantity and quality without recognizing that both are on the same dimension 
I would not go so far as to say that quantitative indicators are qualitative 
by their nature, but there is a clear relationship between the two. A sub- 
stantial part of the^qudlity of a program can be de,monstrated objectively. 
h)r example, publications in referred journals are posited as indicators 
of quality, but their frequency can be detemineu objectively. The numbsr 
of faculty with terminal degrees, the size and diversity of facilities, the 
level of faculty research activity, the academic ability. of students, the 
fittitudf? of students and employers concerning the program-- these are all 
indic.itors of q.udlity objectively assessed. Other aspects Such as unit 
co',ts, ':.tudent credit hours, student faculty ratios, classroom hours, and 
productivity are also indicators of quality in that they measure effec- 
tivpne'is an'' ^ficien^y. It has always struck me as ironic that the 
itr.ti tutions criticize our agency for relying as heavily as we on 
quantitative indicators, ignoring the fact that in institutions tenure 
decisions, nromntion decision^., merit pay decisions are all based on 
'jUiititi tati V.' considerations, if wo can recognize th.st we are into matters 
of sut)% tani.e . 

Wa.iii'iqtnn has di/idod program nc^d into three subcategories, sorrte of 
whi(n are pcrtinrnt to thp state agency and others to tiie institution. The 



94 

9 



three ureas are student need, society and manpower need, and faculty- ' 
institution need. Student need is reflected in enrollments, admissions, 
applications, and graduations. Educational need is apparent in the neces- 
sity for offerings in one field to complement the offerings in another. 
. This is an argument for otferings in a related field, thojgti'not neces- 
sarily for programs. A program in physics at the graduate level is riot ' 
necessary to support a chrrnistry program, although graduate-level courses 
may be. Faculty or institutional need tends to be expressed in subjective 
terms--and the need itself also seems to reside mainly in that realm. 
Faculty need is described as the professional rewards' faculty members obtain 
from working with graduate students, doing research at the graduate level, 
and external professional recognition gained from being involved in graduate 
progr'dms in their field. ^ 

In assessing these four dimensions, there are some differences in the 
perspectives of institutions and state agencies. In general, our agency 
wiil focus on student and sosietal need requirements, as the data necessary 
, # to make judgments in these areas are often available from extra-institutional 

"sources. The institutional initiatives can be effectively directed to the 
development of the case for faculty and institutionai'need. Information 
necessary to document these areas is found within the institution, although 
not usually in published form. 

Relateji to cost is the overall interest we have in program efficiency 
and effectiveness. These two words, efficiency and effectiveness, have 
essentially the same root, but usage has created connotat>^(gJi>rferences that 
re^^uire some explanation. Efficiency descri bes . the capacity of a program to 
operate with a mi nimum of effort, expense, or waste. Effectiveness relates 
to the ability to direct that efficiency toward a desired end--in our case, 
tlie^education of students and areas of public and student need. 

.vu?n -e examine new and existing programs we direct a great deal of 
effort to determining cost. In Washington we now have an ongoing system of 
uMl cost study, We know for each category, of program in each major field 
what the cost allocation pattern is. We also managed Ij iriove from unit 
:ost 1 nformation--whith is somewhat general i zed--down to program specific 
information. As a result, we were able to go to the institutions with pro 
yram specific cost i nformati ..,i v/hich could then be broken down into unit 
aroa', for compc-'rison across institutions. Since we were using tnese cost 
fi'jures solely. for comparative purposes, we could have just as easily used 
widtjets or nooks. Program A may involve 1.5 times the amount of resources 
.V, (trosjrairi B. We got into some difficulty at the end when people said, 
".Now tliat you nave terminated all those programs, where, are the dollars 
as'.o( iatod with them?" At that point v/e had to make a strong pitch for 
int(_rnai real locatiofis to support remaining programs at the institutions. 
We did that successfully, and in no ca' e did the legislature cut institution 
bud'jets or derrwnd payment of that inoney. 

In our our six-year plan includes institutional role statement^. 

Tncy ,in^ general]/ descriptive, but they still need greater precision. We 
:..iV(' (ic'.c.rifKHj our i n'. ti tut ions in te.Tns -that, in general, accord with the 
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Carnegie classification.; the University of Washington is classified as a 
Comprehensive Research Doctoral Granting University and Washington State 
University is a Coinprehen". i ve Land Grant Researct> Doctoral Granting 
University. One of the stronger institutional arguments we encounter is 
that a given program is necessary to the well-being of the institution. 
It cannot be a "Comprehensi vl- Doctoral Granting University" if it doesn't 
have programs in this particular field. To deal with that problem, we took 
the ninety-one flagship institutions in the United States identified by the 
Carnegie Coinnission as Research I and Research II doctoral granting univer- 
sities and, through a rather laborious process, looked at all their program 
offerings. We categorized them finally into 152 different graduate program 
areas, and we found that there was no single .graduate program that appeared 
in the inventory of every one of these institutions. From that we deduced 
that there is not a single graduate program essential to an institution's 
status as a com^!rehensi ve university. There we r^ some programs that showed 
up more frequently than others, however. We^found that' there were sixteen 
proyrahc areas in the graduate offerings at three-fourths or more of these 
institutions, .ie found another fifteen programs on the inventories of 
the next one-fourth, allowing us to conclude that thirty-one program areas 
wer^^ offered by iiore than half of these flagship institutions. We were 
really looking for ^ore curriculum at the graduate level, although we 
ended l)y defining what it was not rather than what it was. The core cur- 
riculum lUGStion persists, particularly at the undergraduate level, and it 
IS an important consideration for institutions. If they don't define the 
. core curriculum at. the undergraduate level--what it takes to be an adequate, 
responsive college or ■university--then I suspect that somebody's going to 
\ do it for them. It is a job that "has been put off too long. 

We nave found that the presence of our agency within the state has 
tak,.>n d tremendous amount of pressure off the institutions and the legis- 
Liture. A number of (lues.tions that used to occur every session are no 
longer being a^kod. The legislature doesn't criticize graduate education 
in Wash HI') ton ,3ri ^ longer, because they know that these programs have been 
reviowe.l liy the C;^iincil. By the same token, when an institutional budget 
request '-.omes in for a new program that has been recommended by the Council, 
It is invariably funded by the leoislature without much question. Perhaps 
norv significantly, in recoi.iniendi ng a program we become advocates of that 
[)ro.]nri i.efore the legislature; we appear and testyy in its behalf. 

1) if riwiew. have stimulated very effective internal reviews, if for no 
'ither rva ,on ttiar) to preempt the state agency from doing them, v/e now have 
intcM-nai rpviev/ in'ocedures in all of the i ns ti tuti ons , in two or three cases 
tht>y dr-.^ vpry qood. As a reoui r. nent in our degree guidelines, we also 
in',i ,t th.it, iipw [irofjrams bn (>valuateJ at the end of their fifth year of 
i ■eratiijii, As a rrsult, institutions do not turn their attention away 
ffoiii a proqrain on e it has started. They k'low that at the end of five 
/ears trie proqrar'will [)p reexamine.' and thil it will le terminated if it 
'-nasn't measured ur) to its iriginal expectations. The Council will terminate 
oiu\ 'yiuj] [)ri)qrari fiexi. week. 
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Not only have 'reviews stimulated effective institutional level reviews, 
they have also stimulated more effective departmental latel reviews. De- 
partments now spend more time evaluating program structures and purposes, 
looking at departmental goals, and setting priorities among those goals than 
was ever the case before. The quality of programs has also been improved. 
iJe review approximately fifteen new program requests a year; ten years ago 
institutions were instituting four or five times that many. Very few pro- 
grams survive the internal review process and get to' the Council now because 
there's the feeling that the Council will reject an inferior program. 

We have also improved the data base on which institutional decision 
makers make their judgments about programs. We now have in Washington a ' 
consistent format employed throughout the review process. After the de- 
partment has a general approval from the state for one of its preliminary 
proposals, it can proceed immediately to collect a^smropiriate data. Not 
only do the Council and Council staff have access toNthat data, but also 
the institutional decision makers. Nj 

I think we have improved inter-institutional coimuni cations. All new 
programs have^to be reviewed by sister institutions. We have stimulated 
internal resource allocations by requiring evidence in nevv program proposals 
that internal reallocations have been considered. 

Agencies such as the Council' play a black hat role with respect to in- 
stitutions, both in new and existing program review. We've often taken the 
heat off an institutional administrator by, in effect, putting the pressure 
on the institution to do Something about a program. So he can say, "Look, 
I can't do anything about i'ft . . . the state's pressing me." I do think 
that the .institutions and tWe state live in a tension relationship, a dy- 
nd(,]ic balance, and I think^hat is desirable. The state agencies should 
not become advocates for-fne i nsti tuti c/ns any more than they should become 
dd/ocates for the state legislature vi4-a-vis the institutions. This 
tension relationship has had salutary effects in the state of Washington. 
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INSTITUTION LEVEL PROGRAM REVIEW 
Hugh Petrie 

I will give an institutional perspective, that of the University 
of Illinois at Urbana-Champaign , on the question of how state and system 
level review processes can promote and strengthen institutional practices 
of evaluation and review. What I would like to do is talk about two major 
are^is. One is how, from my perspective, these processes vary by level, 
and second,- what the results are by level. 

First, it seems to me that institutional review either is, or at 
least ought to be, closer to a disciplinary or expert review than is 
the case at the state level, if there should be encouragement of effec- 
tive institutional review, it would take that difference into account. 
The reason I say that is that one of the problems I have at the insti- 
tutional level is the physicists, for example, telling me that only 
physicists can review physics. And there is a certain element of truth 
in that. It is not the whole truth, because it is not just physics, 
but physics as it is practiced and taught at the University of Illinois 
a^ Urbana-Champa'ign, and that involves a slightly larger audience. But 
i't does seem to me that in one sense only physicists do know where physics 
is going, what the new directions should be. If you broaden this just 
a little bit, I think'it is much less likely, for example, that on a 
state ur system level reviewers would know what rrew academic directions 
to encourage or promote, which are not simply broad social concerns of 
whether we have enough doctors, or lawyers, or what have you. So, in 
terms of the interesting new academic directions thafmight result from 
institutional review", those tend to be discipline oriented, and, by 
and large, -I do not think that the states and systems promote those 
perhaps as well as they should. 

, That is not surprising when you come to the second point, which is 
that a state is a sort of geopolitical entity and within that ^geopol i ti cal 
efitity are* very different educational entities.' Community colleges are 
•much different; from regional colleges and different from research 
institutions. So that wh'en you ask for a state level review (and some- 
how or other that is connected with this geopolitical entity called the 
state), you are likely to get different kinds of questions asked than 
would be asked by the different educational institutions within the 
state, with their differing roles and missions. 

Let ine mention a few examples. What will be existing programs for 
.! mature state university may well be new programs for a new institution. 
So if the state reviews new programs in one way and existing programs 
iri a different way, the "same" program will receive a different kind of 
review iri on.^ set of institutions than it will in anotlier. Consider 
mother example, the problem of duplication. Even with outside experts, 
it is difficult to know wiiether a program in English in one iastitution 
duplicates a program in [iiglish in another institution. The expert 
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review is likely to focus )n scholarly productivity and is unlikely to 
examine the possibility that an English program at the major researc^^ 
university might have quite a different function than the English pro- 
gram at the conTiunity colleges. Cost studies seem also to vary by the 
kind of iristitution being examined. We in Illinois, for example, end 
up wi th -fiii rly low costs because as a major research institution we 
employ large numbers of teaching assistants. Other institutions have 
low costs because they employ part-time teachers. So again, the dif- 
ferent kinds of institutions require that different kinds of questions 
be asked of them. Is it a "good thing" to have Itwer costs because you 
us.e large numbers of graduate student teaching a^istants or because 
you use part-time instructors? From the' state-1 evel perspective, the 
low costs may be the predominant concern. For the individual institu- 
tion, the reasons for the low costs may be of primary concern. 

It seems to me that two things are absolutely crucial when you are 
speaking abf ut evaluation. One, they are always beihg done for some 
purpose or other, and purposes vary for different audiences, for dif- 
ferent regulatory agencies, for institutions. And secondly, evaluations 
are always basically qualitative, alway^ made by the judgments of people 
who do that kind of activity well. If you want to call such people 
experts, all right: the crucial tiling, then, ^is who the experts are 
for- the particular purpose. The physicists are no doubt the experts 
for judging good physics. Who are the experts for judging how good 
the physics department'i s at Urbana-Champa gn? Who are the experts, 
for judgiMcj how much effort we ..should put into business' education? 
Who are the experts for judging the instructional worth of different 
kinds of i n: t i tut ions-? Oftce- ycui_real i ze that the peer groups with whom 
you ought tc be comparing these people might be very different, then 

think th.if the way in which you can strengthen institutional review 
on the sta*-.e or the system level is to try to take account of that 
t'.ic^ .tnd not suggest that there is only one singl e 'purpose , one single 
peer group, one single m,ode or format for cjQ.ing this kind of evaluation. 
I .1111 sympathetic io the state arid system ■l;*vel people because I get 
e.^'icMy thesf s^ime criticisms from the different departments within 
'h^] institution. I am told constantly that I do not recognize the 
di ^prencps and the uniqueness between the departments. And to some 
^x^p'l^ that is true. And the response is, well, let us find the 
peer groiii) for the evaluation thnt we are trying to do and corisult 
fh.it pppr- 'iroup. 

f'prh.ips nothcr way of making my point concerning the d-! f ferences 
in proqr.in; rryjew at different levels is to note that what m^kes an 
.ev, ilu.it iori credible varies from level to level. For the physicists, 
only physicists could iiuike the process credible. State level people 
w<.)uid w.irit some "unbiased" judgmetit. After all, would physicists ever 
reconnend doing ,iw<iy with physics departmerits? At the institutional 
level we hrive to be coricerned' thdt. both state .uid faculty view us as 
( r ed i 1)1 e . 

J" 
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Let me turn to the second general area that I wanted to spend just 
a little bit of time on, and that is, what do the results look like by 
level. One presupposition is that program evaluation will affect budgets. 
At^ Illinois we do not believe that we have> had n^uch, if any, direct ef- 
fect on budget, but, in fact, there is a fairly >ignificant amount of 
indirect effect on bulget. You simply do confirm (or else contradict) 
certain uel iefs held\by the administration about the quality of programs; 
sometimes the budgetary implications can go either way. A strong program 
may or may not get tudgetary help; a weak program may or may not get 
budgete.r7 help. A let will depend upon the* judgment of whether or not 
the strong or weak program is central to the kind of university in 
qliestion, 

A second local effect has been that we have stimulated a number of 
preemptive st^n'kes by the units being evaluated. When you know that you 
cire coming up for evaluation, the indirect effects are simply enormous. 
We had a possible lerger of ^two departments in the College -of Agricultur'e 
in our mind and wanted to talk -to the dean about it. By the t'ime he got 
in to talk to us about >t, he had already set up a committee to look 
into it thereby effectively precluding any direct recommendation from 
us. So, the nroblem of timeliness needs to be thought of in a fairly 
broad context, because timeliness not only refers to the fact that an 
evaluation is done, a recommendation issued, and a decision made, in 
that order. tV\ evaluation may be very timely even if it nas not taken 
place yet, because a good administrator'/ill know that it is going to 
t<ike pi .1^(2 .]rid use it- in, one way or another. I think we need to remember 
that when Ve a'^^e thinking about the effects of evaluations. 

We have also had some effect on general acadejni'c good sense. Even 
at a rejat:ive1y mature university like Illinois,- there simply were de- 
par^hTients before the first cycle of evaluation that behaved very badly 
arid I think perhaps that they behave a little bit better now with re- 
r]!' ! to •■,hpir' f. acuity and ^.tudents. 

-liis' v. the depart!i»ents doubt that we at the campus level fully 
H^proi.^^tf^ the uni(|ue contribution of each department, so, t,oo, do we 
it the LU'ipus level have our doubts that those at the system level 
hono^.M;/ r^pco'^nize that whatever the prestige of the University of 
Illinois, It is due largely to IJr^baria-Champa i gn . We are ^\ot sure they 
Millv ip[)r(M-i itp that. But that is natural and' to be expected. Every- 
one t)pl MWMs th^t his uniqup stiH^iqths are not appreciated and appro- 
[uMa^elv r^pw<irdpd. Yet every higher level of organization has to make 
■>on)p Loi'.f.) o J M VP .judfjmpnts, ,iri(j if everyone r^eally were unique, those 
ju^t-ji-»^fi t would [)p ar1)itr^n^y. 

»'n Ml > PspfM t to results O' sttttp level, I think what is happening 
IS a l)it (.Ipor^pr^ but no morT ' 1. We have what seems to be bot^tofii ' 
1 m", mcrpr-ental bud^i'M^inq ui y Stdto of Illinois. Ther^e is a strong 
tp(0 I'l'i for' "ar r^oss-the-boardi sm. " If wp are going to have 7 percent 
■>.ilo7 in<^^o,isos, (.everybody is noinq to qet a 7 percent salary increase. 



100 



It does not look as if progr-am review has made much difference in the 
state, with one possible exception, and that has to do with wh^t are 
called New and Expanded Programs. It looks as if maybe-4:his year 
for the first time there will be some money ai the state level in ..hat 
category. And it may be--dependi ng on how you add up the numbers or 
how you categorize them--that the University of Illinois as a whole 
came out a little better in tnat area than some of the other institutions 
It is hard to tell, in part, because unlike Washington where they have 
been en ] iged in program review at the state level for a good long time, 
dt Illinois we have been involved at it on an institutional level much 
longer than has the state. Only just this year are we finishifiQ our 
first comprehensive st-atewide review of business -^^ucation; the next 
six months will tell whether it has any effect, fortunately, becau-se 
of the way we approach it in Ill-^nois, our institutional evaluations 
focus on the academic, educational kinds of purposes and these have a 
fairly direct beneficial effect on our programs. Since we have also 
been able to sell the state on believing that we have a credible proc- 
ess whose results can be accepted, we have not had the situation where 
we would have to rely on the state's efforts to see a program improve- 
mefit benef'it for our efforts. That may or may not change in the next 
year or so. 

One fin..!l comment. We have hr^d on^ statewide review of non-academic 
programs finished and one is in tfye process of being developed. A year 
or 50 ago the Equal Opportunity P'rograms across the state w^re reviewed-- 
apparently with virtually no comparisons possible. As far as weXould 
tell absolutely rio results came of it all. The second one is the attempt 
to *^eview On the statewide level research centers across Illinois. 
' ^ ofie, again, bothers us with regard to the appropriate peer groups, 
because we have research centers whose appropriate peer groups vary 
from other :ational labs, all the way down to very local, regional kinds 
of research centers whose appropriate peer groups would be the local 
ror^inuoify utilizing the services that they render. And we are terribly 
concerned that we are going tp have the whole bunch lumped together 
'V\6 somehow or other a simple-mirided question asked: What is the return' 
on the state's research dollar for these very disparate kinds of research 
activities? But. fortunatel y , in planfiing for these non-academic reviews, 
the state agency is utilizing institutional input in the planning and 
'iesign process, so we hope that maybe we will be able to convince them 
^o diversify a bit. This is, ifi fact, my retonmiendation in these remarks 
that states cari positively affect institutional r^eviews by trying to 
diversify reviews through the use of appropriate peer groups. 
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SYSTEM LEVEL PROGRAM REVIEW- 
Anthony J. Moye 



The California State University and Colleges System is a nineteen 
campus system. It is statewide, from San Diego to Humboldt (about 
1,000 miles), has nineteen presidents and one Board of Trustees, no 
tuition, 300,000 students, more than 13,000 faculty positions, 16,000- 
17,000 faculty individuals. I think that^you can see that program review 
has considerable consequences in a system of that size. Our aim at 
the systemwide level is to assist in making the program development process 
as realistic as possible and to assure that decisions made in this are^a 
are well informed. 

In- the California State University ar>cl Colleges System, we are 
involved in several levels bf "program review." (It is important to note 
that by "program" I refer exclusively to degree programs rather than to 
options or concentrations.) These' levels include^ 

1. Review and, approval of new programs (academic master planning) 

2. Review of program areas (statewide) to make policy recommendations 
regarding future curricular development 

3. Review of existing programs (campus level) 

/ ' ' #■ 

'4. Review of programs "identified" on some basis as likely candidates 
for elimination (system and campus) 

Part of the problem we currently face is the question of the validity 
of the various approaches we utilize in program review. These processes 
have undergone close examination during the past year as part of the "fall- 
out" of the passage of Proposition 13 and the governor's additional one 
percent cut in budgets. It might be useful to read a sentence or two from 
the report of the Project Team on Academic Programs to give an idea of 
the issues that currently face systems and campuses in the program review" 
aren: 

"One of the consequences of the present level of uncertainty 
regarding budgetary support is the creation of a climate wherein 
programs and resources are managed on a crisis', basis. That is, 
programs (and people!) are cut back or discontinued in termSj of their 
vulnerability to fiscal necessity rather than t^an through careful 
planninq." 

Perhaps an even, more important statement is: 

"In order for a planning process to have any reasonable chance of 
success, each campus and the system must have a reasonably stable 
basis on which to project levels of resource support." 

This is currently a real problem in California. 
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Revi ew and Approval of New Programs 

_ A formal academic master planning process was inaugurated in 1963. 
The process involves a continuous five-year cycle, updated annually, which 
attempts to relate resources to degree programs projected. Thus a new 
.program first appears as a project for future approval. Ultimately, new 
program proposals are submitted to our office for review and approval: 
no program may be started without our approval. 

The policy base for this planning process is a comprehensive Board- 
li statement originally approved in 1963 which is rather broadly 

stilted. It leaves many policies open to interpretation by the staff 

For. example: 

1. "Curricula are to reflect the neeijs of the student and the state." • 

2. "All colleges cannot be all thinas to all people." ' (Curr'icul-a 

in the applied fields and profe'^sions are therefore to be located 
in a systemwide pattern which will achieve an- equitable and 
educationally sound distribution of programs throughout the state.) 

3. Specialized, high cost pr'ograms are to be allocated on the basis 
of review and study of the individual subject areas. Other policy 
statements define foundation or "core" programs as basic to the 
offerings of all campuses. 

Pro cedures for Planning Future Curriculum Development 

Responsibility for implementing Trustee policy with respect to systemwide 
a-rricular development is delegated to the Chancellor. The campus Academic 
Master Plans are submitted annually by each campus to the Chancellor's 
office, where- projections suggested by each campus are reviewed -individual ly 
and in i.he context of the campus' total offerings and projections, the 
offerii^gs of tre system, and where applicable, the state. They are also 
reviewed in term', of campus resource capabilities. Following this annual 
review and .updati ng , the plans are submitted, collectively, to the Board 
or rrustees. Trustee endorsement of all degree programs on the Academic 
Master .Plan is required before projected programs tan be submitted for approval 

Because the planning policies of the Boar?! are quite general, they can 
be applied with whatever degree of flexibility external conditions require 
or individual situations warrant. In determining "needs of stu'dents and 
nteds of the state," for example, there are obviously degrees of accomodation, 
and these are frequently dictated as much by overall public policy as bv 
interna', system policy r j j 

It is important to note that program duplication, per se, is both natural 
and proper in d multi-campus statewide system of colleges and universities 
Duplication is to a certain extent inherent in the definition of foundation 
prbqrams for dll campuses. 
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In summary, a formalized planning process, including rigorous review 
of all proposals for new programs, has been the practice at the systemwide 
level since at least 1963. Proposals forwarded by the campuses for review 
and approval have generally been subjected to extremely close scrutiny prior 
to their submission. Subsequent to the Chancellor's office review of 
programs, they are submitted to ffh^e California Postsecondary Education 
Commission for review and comment, but not approval. 

A final note: Recent and anticipated budget stringencies place real 
limitations on the ability of campus faculty to respond to changing 
societal needs. It is likely that more and more often resources for new 
programs will have to be generated by discontinuing existing programs. 

Only the near future will tell us whether the procedures developed 
sixteen years ago will continue to serve. That is, our.academic planning 
process has been utilized during a period of growth; whether it Will continue 
t9 work a:s enrollments decline is not yet clear. 

Systemwide committees of faculty and administration are charged from 
time to time to study areas where program development is inminent in order 
to ensure orderly allocation of new programs. These Uudies lead to 
policy reconmendations that will guide and limit program development. Draft 
reports and recommendations from these study groups art shared with campuses, 
the statewide academic senate, and other groups beftJre being presented to 
the Trustees for approval. Recent studies have been done in the following 
fields: performing arts, industrial technology, business administration. 
Studies are planned in education, social work, engineering, and nursing. 



R e v i ie w of E xj s t i _n t|_ Pro grams 

I In 1971, the- CSUC Board of Trustees mandated that: "A formal review 
of Existing curricula is to be conducted by each campus as a part of the 
overall planning process." 

• Since 1971, each of the 19 campuses has developed a set of procedures 
designed to review programs on a qualitative basis. Generally, each program 
IS 'scheduled for review every five years, and approximately one-fifth of the 
canlpus programs are reviewed in any one ye!ar, A summary of the results of 
these reviews is provided to the Board of Trustees each November in connection 
with its review of updated campus academic master plans. 

It is fair to say that campus program review procedures range from 
Jndifferent to excellent. Those processes that are most effective involve 
principal campus administrators on an ongoing basis, have institutional 
resources allocated to the process, and utilize the results of the review 
in arriving at resource allocations on campus. Since the reviews generally 
r-'Hiuire extensive efforts by a large number of faculty, they are perceived 
as simply "busy work" unless there is a well-defined process for following 
up on the reports. For this reason, we intend to develop in the near 
future some minimum review guidelines and criteria at the system level to 
provide assurance that quality levels are being judged from a reasonably 
unitorni perspective.! 
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In addition to this formal program review process, each campus has 
developed an informal approach to review. This takes the form of .an annual 
.allocation or reallocation of resources to academic schools and departments 
(at the campus level). The resource allocation process, in effect, carries 
with It an implied program review. When overall resources are increasing, 
the relation of resource allocation to program review is much less apparent 
than It IS when allocations must.be reduced. 

uM-fh question facing each campus is how to inform the allocation process 
. with more than statistical data. Specifically, how are individual campus 
and system goals relating to student access and program quality to be 
incorporated into the research allocation process at each campus? Is there 
an Identifiable poinf at which decisions need to be made about retaining 
program^ which can no longer be supported or about acceptable levels of 

I ^^"^^ ^ °^ deciding within the resource allocation process 
whether certain programs should be discontinued? The most> obvious way'of ^ 
tTrlT. ""'l^ J^"^ questions is through combining the resource allocation 
process with the program review process, either in its current form or in 

roco V- sPeci^i'^lly designed to relate program review findings to 

resource allocation questions. 

The relation of mission and goal statements to program review is so 

PrlZl \ll'lnnU^""'i '^'I't ""^^^^ °^ Academic Program 

Project Team report referred to earlier, we have asked each of our campuses 

that have not already done so to develop specific mission and goal statements. 

Discont inuation of Existing Programs ' , ■ 

. the centV^L^ffice is ambivalent about playing too extensive 

a role in the review of fisting programs (i.e., it should be largely a • 
campus responsibility, ^Ibeit^not exclusively so), the collegial nature of 
campus governance is such that there is often a need for outside assistance 
n arriving at decisions to discontinue existing programs. In general, the 
ess threatening the "regular" program review process is, the more effective 
It IS likely to be in maintaining or enhancing quality. Therefore, in 
JouJlwT H A . ""Sruent with ongoing program review, we ask campuses to 
review (and defend) programs that have been identified as low productivity 
fnf ■ /dentification is made on a quantitative basis-number of degrees 
produced, student facul ty , ratios by level, number of courses and sections • 
ottered per term, number of low enrollment courses , -etc. Programs so 
Identified require "extraordinary justification" in order to continue. 16 
date, our efforts in this area involve "persuading" a campus that it is 
in Its best interest to terminate programs not fully justified. Three or 
four programs are terminated each year on this basis. Clearly, we need to 
do much more in this area in order to be fully effective. However, we 

^r^^Vca^pus flVnl'''^'''' '''''' ' '''''' cooperation 
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Conclusions 

The- entire thriJ3t of our program review efforts at all levels is to 
assure tfiat program decisions made are both (1) as informed as possible, 
and (2) as rational as possible under the prevailing circumstances. Judg- 
ments must be made; they should be made well. 

A major problem with review efforts in a period of declining resources 
is their tendency to foster a spirit of negativity ^nd hopelessness among 
the faculty. I believe that at the system level we need to take great " 
pains to continually reinforce the positive aspects of review and not 
discourage responsiveness to new needs, innovation, and creativity. We 
need to find some way of providing incentives to those who perform wel'l . 



Additionally, at the system level, we must find ways to reinforce and 
support campus administrators and faculty who make difficult, often impossible, 
decisions. For example, when a president decides to terminate or severely 
curtail large programs of indifferent quality in oi^er to provide resources, 
to start new programs, and, in 'the process, gets attacked by students, *^ 
faculty, and legislators, we must. assist in assuring that fair processes 
were followed in arriving at the decision. In higher edujiation, it is 
often easier to make "no decision," i.e., pro-rata cutbacks, a little every- 
where. This approach is destructive of institutional integrity in the' 
long run. In the short term, it undermines the intent and credibility of 
the progrni review process. 

Finally, though I believe our process is ordenly and structured and"' 
informed by data, I would hardly characterize it as "scientific*." Rather, 
in the political environment in which we find ourselves, I believe that 
we in the California State University and Colleges System can justify our 
program decisions on a rational basis. 
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STATE-LEVEL INFORMATION BASE PROJECT 
Roger Bassett ' 

NCHEMS has been involved in the State-Level Information Base Project for . 
four years, thanks to the support of the Kellogg Foundation, which provided 
us with a four-year grant of approximately a half million dollars to study 
the kinds of information required to support hig/ier education planning. About 
a year into the project, the National Center for Education Statistics also 
joined. They had a number of interests related to the state-level informa- 
tion system design function, including the obvious question of how federal 
data collection is related to the information gathering and reporting activi- 
ties of ^tate agencies. They were also interested in developing planning 
functions descriptive enough that each of us' who stand to be affected by 
federal policies and federal grants could begin to make more sense out of 
them and to have a greater input into how those decisions are made. Conse- 
quently, the NCES grant has included an activity called the Federal Data Core 
Project. I am not going to spend time on that this morning, but you will be 
hearing more about that from NCHEMS. It is an effort on our part to help 
the federal government, particularly the education function of the federal 
government, take a look at its planning responsibilities and the kinds of 
information necessary 'to support them. What we have come up with in the 
State-Level Information Base Project is a set of planning guidelines for 
people involved in either evaluating or designing state-level information 
systems. We make a distinction between data and information, and further, 
between a data base approach to the maintenance of data and information 
and the one-time data collection activities of the sort we have been talking 
about these last few days. We are not attempting to prescribe data collection 
of any kind, but merely to guide the kind of planning process used to make 
those decisions for yourself. Butespecially we're not trying to get into 
the business of applications of specific kinds of information, as is typical 
of requests for approval of new programs, and we're not trying to get into a 
kind of comprehensive description of all possible information involved in 
state-level planning either. Primarily we'll look at the organized data base 
aspect of all this. 

There were eight pilot test states involved in the project. Some of 
them are represented in this room, but I want to mention them because we 
put a great deal of emphasis on sharing the experiences of those pilot test 
states, both in the documents that are forthcoming and through activities we 
have developed" which would actually make it possible for you to be in touch 
with people in those eight states. These eight states are in a very real 
sense the resources from which we've drawn, and we hope others will be able 
to draw from them: California, Hawaii, Illinois, Kentucky, New Jersey, South 
Carolina, Virginia, and New York. There were also a couple of states involved-- 
Idaho, which was looking just at the information suggested for state-level 
planning in the area of adult and continuing education; and Nebraska which was 
involved in the adult continuing education. Rhode Island and Hawaii are the 
states that were really attempting to develop ways of using outcomes infor- 
mation at the state level a&> part of planning. 

Throughout SLIB we have maintained a heavy emphasis on the uses of 
information, I guess, in a nut'ihell, our advice to people is: if you don't 
need , the information-, don't collect it. That is greatly oversimplified, of 
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course, because there is a very real need to anticipate information require- 
ments as agency planning responsibilities develop. But there is the sense 
that a state agenty should justify its data base in terms of its respon- 
sibilities. Within that spirit we looked at the major responsibilities of 
state planning, the three main ones are: (1) comprehensive planning, which 
includes enrollmen-t planning and mission, role, and scope planning; (2) budget 
review; since, some state agencies actually participate In the development 
ot budget reqi sts and the allocation of appropriate funds; and finally (3) 
program review. It is possible, obviously, to argue that program review is 
a part of comprehensive planni/ig, but as this workshop demonstrates, it is a 
distinct enough activity with enough resources being invested in It that it 
deserves to be treated separately. 

With that kind of background, within the context of this workshop we 
would like to explore 'with you the program review practices of three state 
agencies— California, New Mexico, and Kentucky— and how those Involved use 
information in their activities. 
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KENTUCKY 
Rrenda Helton 



If one reviews th_e minutes of the Council on Higher Education, it 
becomes readily apparent that graduate programs and program review have 
been an issue in the State of Kentucky since the early 1970's. Since 
19/1, the Council has used several different approaches in graduate pro- 
gram review, before finally attempting internal staff review. Let me 
briefly review for you the history of graduate program review in our 
state. 

The first action in this regard was taken in 1971 when a moratorium 
was placed on any new graduate and professional programs except under 
mitigating circumstances. In 1973, the Council directed its staff to- ' 
meet with the graduate deans and discuss an-approach tograduate program- 
study. The subsequent study committee recommended employment oT consultants, 
ine consultant who was retained recommended the formation of a "task group 
which later became one of several in an overall comprehensive planning 
errort. This task group was composed of a representative from each of the 
eight institutions. They had a difficult time in addressing anything but 
the overall issues of graduate, programs and, therefore, made no recomnenda- 
tions on particular programs. In 1976, the Executive Director of the 
Council appointed a consultant panel of four to conduct a qualitative 
review of doctoral programs only. This review involved only two insti- 
• tutions in this state. As a result 6f the consultant panel's reconmenda- 
tion^that a specific role and mission for each institution should be 
developed before program review was continued, the Council staff was 
directed to write the University Mission Statements. The most important 
aspect of these mission statementswas the discussion of a system of higher 
education composed of various parts each with a differentiated function. 
This was the first time that the Council had publicly confirmed that each 
institution was not to be comprehensive in nature. The political situa- 
tion at the time, along with the Council's judgment that the statements 
not be inflexible, resulted in rather broad statements. The strategy 
was to make the statements more detailed and specific th'-ough program 
review. Thus, an institutional statement which stated that selected 
master s degree programs should be offered would become more differen- 
tiated when the master's programs were registered. Yet, this flexibility 
did not preclude new program proposals compatible with mission limitations. 
After the adoption of these mission statements, the staff conducted an 
internal review of doctoral programs with the mission statements as the 
primary focus. The staff recommendations were presented to the Council 
in May and all but one Ph.D. program has been acted upon by the Council. 

In the midst of all this activity, other changes were taking piece 
in Kentucky which had a direct bearing on protjram review efforts, and in 
fact, set the stage for the approval of staff recommendations on .doc^-orai 
proqrams. The Council adopted a definition of degree program in 1976 
which was directed towards uniformity. This definition has allowed the 
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staff to compile an inventory of degree programs from the way institutions 
choose to describe their degree program offerings. The Council also adopted 
in 1977 the registry of degree programs. To be placed on the registry, 
programs must be approved by the Council and, if not on the registry, 
programs are not considered for funding, or for planning purposes. In 
July 1977, by Executive Memorandum of the Governor, the Council was given 
additional program authority at all degree levels and other responsibil- 
ities which strengthened certain of the Council's statutory powers. 

I have reviewed this chronology of events not to show that the 
Kentucky Council on Higher Education has foundered in trying several 
different approaches to program review at the graduate level, but to in- 
dicate the very sensitive political climate in which program review and 
planning take place. T^q, institutional representative approach to pro- 
gram review failed to work because the representatives were reluctant 
to approach a statewide review and divorce themselves from their own 
institution's future program offerings . Secondly, until certain Council 
responsibilities were put into precise statutory language, the institu- I 
tions had- been fairly successful in lobbying in their own behalf. As / 
an example of the factors involved which are not solely educational, 
let me describe the situation as it related to the doctoral program 
review. 

• The two institutions in Kentucky which offer doctoral programs. 
University of Kentucky and University of Louisville, are by statute 
recognized as the principal statewide university and a major university 
in an urban setting. The University of Louisville came into the Kentucky 
system in 1970 and is located in the most populous area of the state 
with the largest legislative delegation. Both institutions have pro- 
fessional programs in Law, Dentistry, and Medicine, as well as Medical 
Centers which vie for state funds. In addition, since the University 
of Louisville had joined the state system, it had shown rapid growth 
and had received substi?ntial state funding not only for reduction in 
its tuition but also for capital construction. In many discussions 
after the adoption of the mission statements, it became clear that we 
had two doctoral institutions, one with comprehensive offerings and one 
seeking to be comprehensive in its offerings. Since one institution 
was in a more populous area and had a larger legislative delegation, the 
staff's recommendation fiad to be soundly based on educational practices, 
statewide concerns, and unnecessary duplication while recognizing the 
sensitive political environments. 

The staff's preliminary work began with the development of an 
overview paper which described graduate education in an historical, 
national, and state perspective. This paper was primarily developed 
for our lay Council, but it also described the review process and the 
dimensions of program review. At the same time, the. staff wrote sum-, 
mary reports of four to five pages in length on each of the 64 doctoral 
programs at the two institutions, 51 programs at UK and 13 at UL. These 
'summary reports were intended to be. an objective description of the pro- 
;rams as presented by the institutions in their program report submissions 




to the Council. The summary reports each had the same format: program 
description, student data, faculty data, need assessment, and summary 
or analy,sis. Next, the staff developed four criteria in priprity order 
upon which our recommendations were based. These criteria were bc.sed 
on sound educational principles and did not consider politics. 

The first criterion is the state's need for- the program either in 
terms of manpower needs, research needs, or public service needs. The 
second criterion is the relationship of the program to the institution's 
mission. The third criterion is adequate student demand for the program 
based upon an average of two doctoral degrees conferred over the five- 
year reporting period. The fourth criterion is evidence of quality con- 
siderations, i.e., resources to support th6 program. As a result of the 
prioritization of these criteri a ,. programs which did not meet thie first, 
criterion were no longer considered. Many programs met the first two 
criteria, but did not evidence adequate student demand. 

Programs which met all four criteria were recommended for registra- 
tion, 29 at UK and 1 at UL. Programs which met the first two criteria but 
did not meet the demand criterion were .registered with comment, 18 at 
UK and 7 at UL. This category was developed because the staff felt 
the criteria should be adhered to and that it would be a disservice to 
programs which unconditionally met all the criteria to aggregate them 
with programs which did not. Secondly, thi s '^category put the institu- 
tions on notice that these underproducti ve programs would probably be 
reviewed more critically in the next review cycle. 

The Council on Higher Educatioti began developing its Management 
Infomidtion System in 1974. Early in 1975, the National Center for 
Highr- Education Management Systems initiated a State-Level Information 
Bdse Project with funding from the W.K. Kellogg Foundation. Kentucky 
was selected as one of the five pilot test states to assist in the 
development and testing of the SLIB concepts. The SLIB effort helped 
Kentucky to define Its needs in the planning area and begin to structure 
the Management Information System so that data could be fed into the 
system, validated, and utilized in various combinations. 

The major use of the Council's management information system and 
Jit,i base during the doctoral program review was not as intensive as 
utilized during the master's program review. 

However, we did utilize the system to verify enrollment and degrees 
conferred datj. We also used cost study data in the doctoral program 
review and have already utilized it to a greater extent in the master's 
program review. We also made use of stale manpower projections from 
another state agency. We have had discussions with that agency asking 
^hel^ to specify degree levels in their projections. 

The master's program review is ongoing and we expect recommendations 
^.0 bp presented to the Council in the fall of 1979. The baccalaureate 
[iroqrMm review is proqressinq and is much more data intensive than either 
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the master s or the doctoral program review. A summary sheet of perti- 
nent information on each baccalaureate program has been coded and is 
currently computer retrievable. 

The review activities have also resulted in changes in our data , 
base. As a result of program review and other ongoing Council activ- 
ities, the ta base has been greatly expanded. This is one of the 
advantages of tha SLIB project. 

ihe SLIB approach allows for expansion of the data base without 
causing any delay in the use of the existing base. It is very flexible 
and can be expanded by definition of the new item. This, of course, 
saves much time and effort on the part of the programming staff and 
gives rather quick response to the user groups. 

One prpblem we have encountered is the fact that as items are 
added to t^e base the demand for information has exceeded expectations. 
Therefory, it^^s difficult to keep pace with the planned development of 
the bas^aad siitisfy the users who require information from the base. 

WhlXa^st of the information utilized by this agency is computer- 
ized, there are certain areas of information which we have maintained 
on microfiche and other retrievable media. 

One additional action taken by the agency was to establish the 
Kentucky Center for Educational Statistics which is designated by statute 
as the repository for highar education information. Our thought there * 
was to avoid having several sources which could provide conflicting 
data and create credibility problems. 
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CALIFORNIA 
Norman Charles 



One ccndition that i c .important -when we're talking about any of 
our activities in California is the sheer /5'ize of the enterprisi. We 
have, as you may know, 106 community colleges, 19 campuses of the 
State 'Jn iversitly, nine campu-r^es of the University of California, along 
with several hundred private degree-granting school, and colleges. This 
tends to complicate things eW^rmously. 

Each system office is responsible for the approval or disapproval of 
new .programs, and each each of these central offices maintains an extensive 
data ba'se for its own systeni. In the case of new program proposals-, there- 
fore, a proposal is'reviewed at the Commission only after it has made its 
way through a series of review procedures starting at the campus level. 
We seldom deal directly with the campuses. 

■* 

We have developed a form for proposing new programs from each 
segment, based on a standard set of criteria. These are criteria we' 
all use. In fact, it seems to me that if someone wou' d come up with 
new criteria for program review, it would cause the same stir among 
us as the discovery of a new element among physicists. Nevertheless, 
I will mention a few of the standard criteria and try to suggest how 
we use them. 



iter fa for Program Review 



The essential document in any 'state for program review is an 
updated inventory of what programs exist. It took several years' effort, 
but we now have an inventory program in California col'leges and universities 
that' is computerized so that it can be conveniently updated annually. 
We have a 15-page index, double columned, which is invaluable, and is 
t,he first thing we refer to whenever a new program is proposed. This 
year we have drstributed the inventory not only to the campuses which 
responded, but also to high school counselors. 

AnHmportant crit-^ria we use is student demand. Recently. we did 
<j computer printout of enrol Tments in each major program on each campus at 
all levels for the past five years. That will also be added to annually 
so th<it we will be able to see exactly what is happening in a comprehensive 
Wiiy. I find it to be extraordinarily useful as one indicator of student 
(jcmand. I should point out that I'm speaking of only what we have "in- 
house," so to speak. We also expect to get 'the college's judgment in 
earh of thesp categories. In the case of student demand, of course, 
'herp are other enrollment trends over a period of time, anil it is often 
difficult to indicate exactly how many students a College might expect 
tn (Mirnli. I've never trusted the approach that distributes a questionnaire 
to p(>o()]c in.. I related field, asking, "How many of you woul|l sign up for 
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this major if we offered it?" 'Obviously, the use of all these categories 
calls for judgment. ^ 

Manpower information is, of course, another central category of 
documentation, particularly with two-year programs or programs witlr^a 
specific occupational focus. However, we are a long way from making.' 
a science of projecting manpower needs. Although it is a criterion that 
require? extreme care in interpretation, one responsibility of a planning 
and coordinating ageniy is to try to see to it that the number of persons 
trained .in a fieM and the number who can be employed remain in some 
balance. That will never be worked out perfectly, of course, but we 
have come to feel that, for a new Ph.D. program there has to be some evidence 
that a new batch of graduates can be absorbed, particularly in cases where 
there are other programs in the state in the same fields available to 
students. ' . 

Another criterion we use is the total cost of the program. We have 
'lot dune a thorough cost of instruction sti^djf as yet, although I believe 
we are*^ under direction to begin doing them. Instead, we have asked for 
individual prqposals to provide projections of the number of new faculty 
whicn will be required or what facilities and equipment might be needed. 
I think rill of us are aware of the possibl-e misuses of a cost of instruction 
study that would lead us to charge more for certain degrees than others. 
I've actually heard it suggested, only half facetiously, that we might 
begin to base tuition and fees on the major a student chooses. I can 
'imagine a college advertising a special on sociology (Programs for next 
year at $398.00. But that's an extreme misuse of that sort of * information. 

We have listed a category in regard to both new and existing programs 
which has to do with the maintenance and improvement! of quality. X)ur 
approach here has had to be that we foster and encourage programs of the 
highest quality, and yet that we have left that determination to the 
campuses and to segmental administrative offices. There are times, of course, 
when one certainly feels tempted to comment on a proposal in these terms, 
but we try to refrain from doing 1:hat., ^ 

The advancement of knowledge is another criteria for consideration. 
There may be no data evidences for this criteria. Here we have in mind 
those programs and fields--such as energy or certain areas of medicine--^ 
where there are no other programs, we've never enrolled in the field, 
and do not have any evidence of what the job^market is. And yet, it seems 
to us t,hat the program area ought to be enG6iiraged. Obviously, we have to 
rely on the stature of the faculty proposing a program to ensure that it 
is/a field that is legitimate and useful. 

I just want to say, in conclusion, that in the process of program review 
at all levels we need all the information we can obtain. There is no amount 
o(, information, however, that will eliminate the subjective judgment invoved 
in. each decision. We seldom get proposals that allow for easy decision-- 
in other words, high cost programs with no student demand and no job market 
and seven programs already in the state. If we did, there would be no challenge. 
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NEW MEXICO 
Robert Rhodes^- 



Introduction ' 

At the request of the New Mexico Legislature, the Board of Educational 
Finance, in cooperation with the Academic Council on Higher Education (which 
is composed of the chief aca'demic officers of New Mexico institutions of 
higber learning) conducted progVam evaluations in New Mexijjo's six institu- 
tions of higher education. No special funds were appropriated in support of 
this study; each institution, irKcodperatipn with the staff of the Board of 
Educational Finance, undertook tfrvs review as part of the regular operation 
of the institutijDn. 

Significant/ changes in the nature of the American society make it 
imperative that/academic programs be^ systematical ly examined to determine 
if graduates arje^ in fact, being trained to meet the needs of this changing 
society. No si^le institution can satisfy all these new needs. More than 
ever there is aViecessity for matching resources to these growing expecta- 
tions. 

« 

This study was undertaken as the first step in a systemwide approach to 
academic program reviews that will hopefully allow more thoughtful planning 
and mV^ rational setting of institutional priorities. "The following insti- 
tutions are included in the scope of this study: 

University of New Mexico 

New Mexico State University 

New Mexico Highlands University 

Western New Mexico University 

Eastern New Mexico University ' 

New Mexico Institute of Mining and Technology 

.A review of academic programs on a statewide basis offers the opportunity 
for more effective coordinatiorx' of all programs, but the achievement of this 
coordination is directly dependent upon the establishment of an ongoing process 
of interiristitutional program monitoring. This study should be seen as fherely 
a first step in this ongoing review process. 

The specific charge of the legislature to examine the question of 
program duplication led to this comprehensive review of all academic programs 
in order to determine if some unjustifiable programs exist in New Mexico's ^ 
institutions cff higher education. 

*''%he first year's study revealed that there were many programs being 
''(5tfeV'ed that required careful examination to determine if those programs 
should be continued. The study also revealed that a certain amount of 
program duplication is both necessary and desirable. Accessibility of 
college education and the availability of well-rounded undergraduate pro- 
grams are factors which must be considered when assessing the justification 
of duplicative programs. 
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A recent report on long-rc^nge educational objectives identifies 
significant aspects of changing conditions which universities face in 
the next decade: 

• Slew dnrollnient growth to 1980 and stabilized enrollment 
thereafter; 

f Stringency of financial resources for higher education and 

for the universHies at both state and federal ^levels; * 

• A reduced number of academic career positions for recipients 
of doctoral degrees, combined with continued demand for doctoral 
and professional-degree graduates in other types of careers; and 

• Continuing changes in student interest and in socia-1 trends, 
giving rise to new needs toward which the universities' resources 
for teaching and research might better be directed. 

Program Review Procedures 

In order to identify possible deletions in program offerings at the 
SIX institutions of higher education, the following steps were taken: 

• Development of a fact sheet for every program at every level in 
each university. 

• Development of criteria for identifying programs that should be 
subjected to closer scrutiny. 

• Development of a precise procedure for identifying programs that 
should be dropped. 

• Development of a complete- and accurate inventory of degree programs. 

• Development of an ongoing procedure to produce accurate data on 
program enrollment, graduates, and costs. 

• Visitation by the Academic Council of each institution and presen-. 
tation by the institution of actions taken concerning programs 
identified by the American Council for^review. 

Sc hedule for Program Evaluation Stud y 

June - Reviewed proposed design of study with Academic Council'.^ 

- Reviewed study design with Legislative Finance Committee.. 

- Reviewed present approval procedures with graduate deans. 

- Completed on-site consultations with university personnel. / 

July - BEF staff completed detailed degree program inventory "fact 
sheets" 

- Academic CouncM Meeting: 

--Progress rpfjorts on mission and role study 
--Distn-iM'.ion/discussion of degree program fact sheets 
--Rearlied consensus on preliminary criteria for evaluation 
of exi?^ ting programs 
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- Data collection and identification of selected criteria. 

- Institutions valwiated fact sheets corrections by July 30. 

- BEF staff refin^ criteria for evaluation by testing criteria 



against fact sheets. 



August - Completed data collection and applied selected criter^ia. 

- Incorporated n.issing and/or corrected data on to fact sheets. 

- Final application of refined criteria set to validated fact 
sheets in order to develop^ lists of questianable programs. 

October - Academic Council reviewed flagged programs and institutional 

representatives presented initial reactions. Format of present 
reportv agreed upon. 

December - Report delivered -to Legislative Finance, Committee. 
. Selected Criteria 

Criterion One-- Number of graduates by degree level from each program in last 
five years: 

Ba ccala ureate level programs having four or less graduates 
per year on the averag/over the last five years. 



Master's level programs having three or less graduates per 
year on the average over the last five years. 

S pecial i St level programs having three or less graduates per 
• year on the average over the last five years. 

D octor al level programs having two or less graduates per year 
on the average ove/- the last five years. 

4 

Criterion Two-- Average of head count students enrolled in Fall 1974 and Fall 



Baccal aureate level programs having 12 or less majors per year on 
the average over the last two reporting periods. 

level programs having nine or less majors per year oti 
the average over the last two reporting penods. 

5pecj_aJJsl level programs having nine or less majors per year 
on the average over the last two reporting periods. 

Dpaoral level programs having six or less majors per year on the 
average over the last two reporting periods. 

Procj^res s on_ JP rojgrams^ Reviewed 

t 

Progress to date shows 14V.programs failing Academic Council criteria. 
Action has been taken on 119 programs: 71 have been dropped or combined; 
27 ara still under study; and 21 are to be continued. 
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STUDY OF ACADEMIC PROGRAM REVIEW" 



Introduction 

This report is the result of a joint study conducted by Lilla Engdahl 
for the Western Interstate Conmission for Higher Education. (WICHE) and 
Robert Barak, working as a consultant for the National Center for H4gher 
Education Management Systems (NCHEMS). It was undertaken as a result of 
d growing interest on the part of both sponsoring organizations, the higher 
education conmunity, and certain governmental bodies in the evaluation of 
academic/occufj^tional programs. The study's purpose was to develop an 
overview of the current status of academic/occupational program review^ in 
colleges, universities, system offices, and state higher educa/tion coordi- 
nating and governing agencies. The study focused on program review activ- 
ities both within institutiorys and at the state level. This report is 
based on survey responses from state agencies in the western states which 
are included under the WICHE Compact: Alaska, Arizona, California, Col^do, 
Hawaii, Idaho, Montana, Nevada, New Mexico, Oregon, Utah, Washington, ans? 
Wyoming. The results of a national study on the same subject will be 
published separately by the National Center for Higher Education Management 
Sys terns . 

The report includes a general discussion and a report of the survey 
results. The general section includes a review of pertinent literature, 
astate-of-the-art description of program review activities, and an anal- 
ysis of the maior issues related to .program review. The report of survey 
results provides an analysis of responses to the survey; it includes a 
section devoted. to institutional responses and a section summarizing the 
approaches taken to program review by each pf the thirteen western Sudte 
agencies. 

The published literature on program review/approval is very limited. 
The authors are aware of no general studies of the areA-other than the 
present effort. There are a few general works on academic program evalua- 
tion (Dressel, 1972, 1976-, Cranton and Legge^, 1978; Heydinger, 1977); and 
numerous works on either evaluation in general or curriculum development, 
which in oart may touch on academic program review/approval (Wood and Davis, 
1973)'. The elusive topic of "quality" has been a frequent subject of the 
literature.. Several notable attempts have been made to develop ways of 
assessing quality as an important aspect of graduate program review (Clark,, 
1975, 1977; Blackburn and Lingenfel ter, 1973; Kelly, 1976; Grodsky, 1979). 
Other studies have concentrated ocuprogram review at the departmental level 
(Clark, 1977 and 1979; Smythe, et al., 1978). There have also been several 
relevant conference presentations on designing and implementing program re- 
\ view (Minis, 1978) and the need for program review (Heydinger, • 1973) . Only 
\pne paper has concentrated on procedures for program approval of new pro- 
grams at the institutional level (Gilmour and Lee). One-well <Jone unpub- 
lis^l\ed paper presents a model for review and abandonment of programs 
(Gilikpur, et al . , 1977). 
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External prcygram review, perhaps becausie of its more controversial 
nature, has been trie subject, of a number of recent inquiries. General 
studies of extra-institutional program review (Rudnick, 1976) and state- 
level coordinating/governing agency program revie\^ (Barak, 1976, 1977; 
Barak and Berdahl , 1978) provide a rather comprehensive report on the 
status of these external reviews. Two other studies concentrate on 
legislative program evaluation efforts (Petersen, et al., 19/7; Berdahl, 
1977). Program discontinuance is the subject of a current study by a 
graduate student (Melc^iori, ongoing) and a brief recent survey of state- 
level program discontinuance was undertaken by another graduate student 
(Skubal, 1'979). These latter two studies complement another recent major • 
study on program discontinuance (Dougherty, 1979). * 

• ♦ 

Prograin review and approval are of increasing significance to insti-., 
tutions, system level offices, and state postsecondary education coordinate 
ing and governing agencies. The reason for this significance is the well-'> 
known pattern of events characterizing higher education today: ^ 

• increasing costs combined with a loss of support money as a 
result of inflation and recession; 

• reduction in financial support from private foundations and 
federal agencies; 

• diminishing employment opportunities for persons holding 
advanced degrees; 

• an increased emphasis on accountability; 

• the general trend of declines in enrollments and in college- 
going rates. 

Most state-level higher education coordinating agencies have been 
involved in program review and approval since about 1970, although some 
were involved much earlier. Prior to 1970 coordinating agencies often 
had statutory authority to review and approve new programs, but in the . 
last decade statutory responsibility for the reviav of existing programs 
has often been added to their duties, either direc\tly or through budgetary 
action by the state legislatures. Some state agencies which have no statu- 
tory authority have conducted reviews of existing aind proposed programs as 
a coordinating responsibility. In either case, the\ formal review of grad- 
uate programs is more common than that of undergr'aduate programs. 

Program review and approval are conducted for many different purposes; 
however, most of them can be grouped under the general term "management 
tool." At the state level, prfogram review and approval are oriented to- 
ward improved coordination and planning. At the institutional level they 
are focused on "program improvement." Even where financial exigency pro- 
vides the initial impetus for program review, institutions emphasize the 
ways in which the elimination of weak programs can enable them to concentrate 
their resources in other areas. 
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vThe term "program" is defined variously by institutions and state - 
agencies, and "program' and "curriculum" are often used interchangeably. 
However, the most common definition of program is a series of courses 
arranged in a scope and sequence leading to a degree or cer.tificate. 
In some institutions and states this includes majors or fields of 
specialization. 

The following section provides a description of procedures used in 
program review conducted at three levels. It is di'vided into general 
discussions of: (1) institutional program review procedures; (2) state- 
level program review procedure^; and (3) program approval procedures 
involving both the state and institutional levels. 



Institutional Program Review . I ns ti tutiona\ procedures for program 
review vary about as frequently as there are instuutiofis conducting 
reviews. Conseq'uently , the description of the procedures which follows 
is of necessity both a simplification and a generalization of the proce- 
dures actually utilized by colleges and universities.. 

At the institutional level, primary responsibility for program review 
rests with either the graduate dean (primarily at research universities) 
or the vice president/dean/provost. A slight variation to this general 
pattern is in the two-year community colleges where the dean for vocational/ 
occupational education may be responsible. ' 

A comnon method of scheduling program review is to rotate programs on 
a cycle of 4-7 years (5 years seems to be the average). Program review is 
often scheduled to coincide with accreditation visits. Other situations 
can trigger a review of a program, e.g., a poor accreditation report, drop 
in enrollment, or a"change in leadership at either the departmental or 
dean level. One small private institution reported that a review of pro- 
grams occurred whenever there was a faculty or administrative staff vacancy. 

The first step in the review process, .once a program has been identified 
for review, is a departmental /uni t self-study based upon general guidelines • 
provided to all departments/uni ts being reviewed. Some self-study guide- 
lines are more specific than others, but niost include a statistical profile 
of the department or unit, typically including enrollment statistics, credit 
hours taught, faculty vitae, budget information, and student admissions 
data. * 

A second common feature of institutional reviews is the establishment 
of a review conmittee. The composition of the review conriittees varies 
widely, but frequently includes outride peer consultants, faculty outside 
the unit, and appropriate administ tors. Some commurUy colleges use 
their existing advisory conmittees t each program; however, there are 
mixed reactions to using the advisory committees in this way. Outside 
peer consultants seem to be one of the most highly regarded sources of 
evaluation of "the program being reviewed, but most persons interviewed 
agreed that they are expensive (average about $500 per program) and that 
it is difficult to select an "objective" team of consultants. 
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( The next step is generally an institutional conmittee which first 
reviews both the self-study report and the consultants' report and then 
makes a reconnendation to the appropriate administrative body. The uses 
of these reports vary widely, from storing them on a shelf to Incorporat- 
ing them into the institution's planning and budgeting cycle. An inter- 
esting method of the latter is the "memorandum of understanding" used by 
Ohio State University and the University of Vermont. This approach sets 
forth the fiscal and plann,ing implications of the review process (Arns, 
1979). ^ K V , 

State Ageticy Review . State higher education agencies which conduct 
program reviews cite sever?! purposes. for their reviews: 

• maintenance of .articulation and coordination among the parts of 
a complex system of postsecondary education; 
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facilitation of planning for postsecondary education in the state; 

elimination of unnecessary spending caused by program duplication; 

assurance that available resources are adequate for quality 
education; 

identification of programs which do not meet minimum criteria 

as a basis for decision to eliminate pr strengthen them. 



the state of Washington, selected program data are reviewed annually 
by agency staff, usually the academic program, finance and research/informa- 
tion officers. Those programs which "do not meet previously established 
minimum standards are identified for intensive review. Institutions are 
notified which programs will be evaluated and are asked to provide specific 
data cibout them to the state agency. In addition, each program is reviewed 
on an established five-year cycle. In both cases, the institutions provide 
data, including: 

program cost; 

faculty workload;, 

class sizes; 

evidences of quality; 

state/national production data in the program; 

dollars saved and quality improvements to be achieved from 
program consolidation or termination; 

student interest; < 

societal needs in terms of state, regional, and national needs; 
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t manpower demand and placement; 

t additional dollars needed to establish a quality program; 
t harmony with the institutional role; 
• institutionar%informatiQn. 



. The state agency may decide to investigate other aspects as it fee*ls 
necesf ary. State agency staff reports to the state board or commission 
usually include both a sunmary report of the program review and recom- 
mendations for Loard action. Typically, the state board has authority * 
to make decisions, but in Washington the state agency delivers the report 
to the institutional board in tl ? form of a recomnendation for action. 
The actions which tnay be recormended are: a continuation of the program 
as it is presently offered, continuation with stated conditions; consoli- 
dation with other programs; probation; termination; or further review. 

In this kind of program review, institutions provide data and informa- 
tion to the state agency, but have limited direct involvement with the 
state stdff. / 

State Agency Program Review with Institutional Participation . State 
agencies which have statutory authority for program review often involve 
institutions to varying degrees in their procedures. In Idaho and 
New Mexico, for example, insti tutionaj staff work closely with the state 
staff in designing and implementing the review procedures, while in 
Californ-la and Oregon state agencies liyinit their involvement to monitor- 
ing the procedures carried out by the iliajor statewide sectors and/or in- 
stitutions. This monitoring consists /if examining the procedures in 
light of the statewide master plans to insure the adequacy of resources 
and also to insure that attention is given to both program quality and 
statewide public interests. ^ 

In addition to the participation of institutional staff on statewide 
review committees, institutions also conduct self-studies. Often review 
committees, like those described above in the institutional review pro- 
cedures, are established to oversee the institutional program review. 

Programs are selected for review on a rotati ng -cycle , by minimum 
criteria jointly established by the state agency and institutions, or at 
the request of either the state or institutions. Generally, a two-stage 
procedure is followed. First, institutional data are examined to determine 
if ^programs meet the established minimums. Information typically includes: 



the number of students enrolled in each of the past three to 
five years, and the current year; 
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• the average number of graduates for each of the past three to 
five years, and the current year; 

t the changes in gross total of graduates. 
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Secondly, the institution conducts a study of prograircS not meeting minimum 
/ criteria, using either routinely compiled data and/or extensive ad hoc 

surveys of students, employers, administrators, and faculty. The statewide 
comnittee uses the self-study report to formulate recommendations to the 
\ state board for action on the reviewed programs. 

Outside peer consultants are used in some states in the second phase 
as advisors to the statewide review comnittee and the state board. 

I Survey respondents reported the following decisions as possible in the 
review of existing programs: terminate; continue "as is"; continue with 
modification; continue with strengthening; probation; or restudy after 
stated interval. For proposed programs, the options are to: implement; 
depy approval; review at a later date; or rewrite and submit again. 

Issues Related^to Review of Academic/Vocational Programs 

Review of academic/vocational programs is a highly sensitive activity. 
Perhaps it is the most controversial task that institutions and state higher 
education agencies undertake, for it involves questions about the effect of 
state agency authority on institutional autonomy; conflicting purposes of 
its use both to reduce cost and to improve quality; and the relation between 
program revtew and external evaluation. The following discussion briefly 
examines these and other issues identified through the course of this study. 

An important issue in program review is that of maintaining a proper 
balance t\etween state authority and institutional autonomy. While insti- 
tutional cklministrators generally accept state agency responsibility to 
collect daja from the institution, they are often Lcncerned about the 
authority pf the state in other aspects of review. In particular, state 
agency review of courses is viewed as endangering institutional autonomy. 
Many faculty feel that external examinatiotk^of either course content or 
teaching techniques is a threat lo their i7i dependence. 

On the other hand, state agency staff are often frustrated in their 
coordination efforts, especially by institutions' arguments that the elim- 
ination of a specific program will not in fact reduce costs. 

The lolution--a difficult one--is to establish accepted limits to both 
institutional autonomy and state agency authority. Leon McCarrey, Deputy 
Commissioner of the Utah Board of Regents, rentes 'that a coordinating plan 
^ for all post-high school curricula was put .into effect in that stflte in order 
to avoid unnecessary duplication. The coordination created a" different role 
for each of Utah's institutions and resulted in a division of labor among ' 
tfiem. McCarrey argues that, "State higher. education must be viewed in a 
broader context than can be seen by a single institution and the associated 
board" (McCarrey, p. 87, 1953). He believes that academic departments at 
an institution should decide the structure and content of a program or.cu- 
riculum witnin the established role of the institution. The central state 
agency should judge how the program relates to those of other institutions 
in the state, its impact on other proyramc, and whether better alternatives 
exist. 
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Another activity related, to the question of autonomy is the legislative 
program audit. Conducted- by coimittees composed of legislators or by legis- 
lative staff, t^l^e versions of program review are viewed with apprehension 
by Institutions. The chief criticism is that persons 'from governmental 
positions not only lack an understanding of higher education but also 
that they tend to make political decisions. Although legislatures in twenty 
nine states have established post-audit oversight committees, very few leg- 
islative audits haVe actually been conducted. As -long as state agency and 
institutional program reviews meet the needs of legislatures ^ there is ex- 
pected to be little activity by legislative bodies themselves. 

Often the purpose of program review is at issue: Should it be used to 
reduce costs or to improve quality? Obviously, the purposes for conducting 
reyiews determine not only the methods used but also the outcomes. ^ 

Fiscal constraints, mandated by vote»»s in California, Idaho, and other 
states dnd by legislatures in Colorado, Utah, and Nevada, have heightened 
interest in the use of program review as a management topi for resource 
allocation at both the state and institutional level. In the past, when 
faced with state cutbacks in funds, institutional administrators have in 
many cases simply made cuts "across the board" in all programs. This 
technique is unquestionably, quick and noncon trovers ial in the short run; 
however, over a long period of time it is likely to be costly. Donald 
Smith, Senior Vice President of Academic Affairs for the University of 
Wisconsin System, voiced his concern that this response would "weaken the 
fabric of the university as a whole" (Smith, 1975) over the years. 
He .urged that program review be developed as part of institutional long- 
range planning rather than as a reaction to a financial crisis. 

Both state agency and institutional administrators surveyed in this 
study cautioned that program reviews should not be expected to result in 
cost savings. Experiences in Louisiana, [iew York, Utah, and other states 
indicate that cost savings- do not necessarily result from program review 
and termination. For one thing, the first programs terminated as -a re- 
sult of reviews in those states tended to be those with low productivity, 
thus, those already requiring few institutional resources. Savings are 
more likely to result from a drop in student enrollment accompanied by 
a .^decreased need for faculty and equipment. 

p- In any case, the isSue in cost savings is basically whether the in- 
stitutions will retain the use of any funds saved, or whether the state 
Willi keep the money. While this question is often seen as another disagree- 
inerH between states and institutions, it is not as simple as that. In re- 
sponse to our survey, state staff showed a clear awareness that, although 
prograiii reviews could help with immediate decisions aboGt where to make 
budget cuts, over the long run reviews should serve instead to reallocate 
funds in order, to help institutions strengthen needed programs and termi- 
nate weak ones. • 

Selection of criteria is one of the most critical aspects of program 
review; obviously, it is closely related to the purposes for which the 
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review is being undertaken. The criteria to "be u-^ed and the weights for 
each criterion should be agreed upon jointly by all participants at both 
institutions and state agencies. Making this decision early in the plan- 
ning stage helps avoid' later conflict and dissatisfaction. Moreover, 
' application of previously adopted criteria helps assure equitable evalu- 
ations for. all programs being reviewed. 

An overemphasis on quantitative data ignores achievement of program' 
•objectives, geographic availability of, and need for programs as well as 
other nonquantifiable data. Programs with different purposes' should ob- 
viously not be evaluated by the same criteria. 

Study participants expressed concern about the need to establish 
criteria for the assessment of the quality as well as the productivity 
of programs. Information for evaluati^ig quality includes ratings of 
degree programs by experts in the field, by deans and faculty outside 
the institution, or by professional association officers. Peer ratings 
are considered by Barak and Berdahl (1978) to De the only consistently 
reliable measure of quality. Mary Jo Clark's study at ETS (1979), however. 
Identified environmental and operational indicators which are also useful 
in assessing program quality. 

Russo et al. ( 1977) expressed concer^n about the use of quanti- 
tative data in making decisions to terminate' programs: 

Head counts, credit hours, full-time equivalents, time to 
graduation, drop-out rates, dollar cost per degree, volumes 
in, the library, and student/teacher ratios are relatively 
easy to compute. Number numbness is what W. David Maxwell 
( 1973) labels the phenomenon,. The criteria used for judg- 
ing "goodness" are selected because they can be counted . . . , 

One problem, according to the Washington Council for Postsecondary 
Education, is that such criticisms tend to separate quantity and quality 
without reco]nizing that they may represent two ways of expressing the 
same factor. They contend that much of the quality of a program can 
in fact be demonstrated quantitatively. For example, indicators of qual- 
ity which can be quantitatively assessed include the frequency with which 
faculty, publish in referc-ed journals; the number of faculty with terminal 
degrees; size and diversity of facilities; level of faculty research ac- 
tivity; academic ability of students; attitudes of students and potential 
employers toward a program. Measurements of effectiveness and efficiency, 
such as unit costs, student credit hours, student/faculty ratios, class- 
room contact hours, and degree productivity are also considered by the 
Washington Council to be indicators of quality (WCPE, 1977). 

No single schedule for the review of programs can be described as 
typical. Continuous reviews (or reviews which recur too often) place a 
great burden on institutional staff, but continuous review is the only 
way problems in programs can be detected soon enough to be addressed 
effectively. Periodic reviews may allow years to pass before problems 
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become evident. Most institutions and states have adopted a flexible 
approach which includes an established cycle in which each program is 
automatically reviewed at certain intervals and, in addition, includes 
criteria which will automatically identify programs for an unscheduled 
review. Some agencies and institutions are searching for ways to select 
programs for review which are not in trouble because of low productivity 
but which may need strengthening for other reasons, such as relatively 
high enrollments and unmet demands. 

Selection of participants is critical to the success and acceptance 
of program review. State agency staff are often perceived by institutions 
as outsiders who lack institutional experience and undei ^nding, while 
state agency staff believe institutional reviewers lack objectivity. 
Institutional staff charge that the state personnel's lack of academic 
experience and their remoteness from the campus weaken their understanding 
of academic concerns. However, training programs for state staff, such as 
that developed by the Education Coiimission of the States, are attempting 
to meet the need for knowledgeable people with both the experience and 
analytical skill<^ necessary to understand the complexities of statewide 
coordination. 

Within institutions program review activities are often resented be- 
cause they are interpreted as an effort to justify reduction of support 
for individual programs. Faculty involvementln review is very importailt. 
Orientation and in-service training in evaluation methodology for faculty 
and staff in institutions can help to improve understanding and partici- 
pation. The support staff should also be represented in the procedures 
and infonned about results, since the termination of specific programs 
has an impact on the need for library holdings and for other services, 
such as student support. Both state staff and institutional administra- 
tors reported in the study that the time used to build cooperation and 
trust among participants is a worthwhile investment. 

Outside consultants are used by fewer than half the state agencies 
surveyed in the study; only two states in the West^Siutinely use out-of- 
state consultants. Cost seems to be a major deterrent'in their use, 
although prograin reviews within institution' tend to include them fairly 
often, particularly in the review of gra^ ograms. 

Selecting consultants is a difficult uuok. Early in the design of 
the program review procedures, the-ije lection process and standards for 
consul tants should be agreed upon both by institutional and state staff. 
One question in the use of consultants is the cost compared to the value 
received from consultant review; other areas of controversy--such as the 
dvaijability of consultant reports to all parties--can be eliminated 
through planning. 

The primary criticism of data collection at the state level comes 
from institutions. Their concern is that data' needs are not identified 
far enough in advance to permit accurate and complete collection. For 
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the most part, data must be collected at the institution; this effort 
can be rnerde easier if routinely collected institutional data are used in 
program review. Unanticipated requests for data impose additional work 
on faculty and research staff . Another data problem is the inconsistency 
of definitions; data provided by different institutions are not always 
comparable. For use in statewide reviews of programs at several institu- 
tions, it is obviously important that statistics reflect the same information. 

Another issue in the jse of data is the fear that quantitative data 
will be used without considering unique differences among institutions and 
without considering qualitative information. Moreover, observers fear that 
an overemphasis on productivity will result in a dilution of quality if 
institutions respond by adopting lower admission standards and easier 
•grading practices to improve numerical productivity. The National Board 
on Graduate Education has expressed concern over application of simplistic 
statistics, especially in decisions to terminate progra-^s. 

A major issue related to the approval of proposed new programs is the 
question of when the state agency should be infomied that a new program 
is being developed. State agency staff believe that thoy should be informed 
before significant institutional riiTie and effort have been devoted to devel- 
opment of the program proposal. Once a program has been developed beyond 
the concept stage, the institution is likely to have faculty and courses 
already established; the state agency then has little leverage in-making 
a decision. Several s^ite agencies require that the .institution submit 
a preliminary proposal describing a prograi.i it would like to offer be fore 
substantial planning takes place, fiew Mexico, California, Utah, and Idaho 
--eire among the states following this "early warning" procedure. 

Althougl-, regional or professional accreditation is considered a 
voluntary process, most institutions agree that there are strong incen- 
^ives to pursue it. llhile some aspects of accreditation are si-milar to 
pruijrdni review, few institutions or' states surveyed in this study reported 
that program review and accreditation were conducted concijrrently. Coopera- 
tion 'jetween a few institutions and regional accrediting .issocicftions is c 
a notable exception. Several universities and colleges in California md 
the University of Hawaii have recently begun to conduct program reviews 
ds an integral part of their accreditation self-study, Arizona officials, 
in exaiiiriirig this relationsnlp, found that cooperation between professional 
or specialized accrediting agencies and program review appears to be more 
ft'dsihiG than coordination with regional agencies, whose interest is prim- 
drily with the quality of the total institution. 

"lot dl1 institutional and state agency administrators agree that ex- 
'iMDdl aqencips and institutions can or should coordinate program review 
dctivities; many believe the purposes are so different that separate reviews 
■ nci ('<■. . r\r/ . While use of acrredi tation visiting teams are Drobably not 
[x)'.^', it/1e in a statewide review uf all programs offered in a discipline area, 
it appears from this study that soirie aspects of institutional ^;elf-study 
..in tip u'.eit for hoth program review and accreditation. 




O utcomes of Program Review 

The immediate outcome? of program review are directly related to the 
purposes for conducting thfc reviews; however, there may be unanticipated 
and longer term results a/ well. For example, as an institution states its 
programmatic goals and objectives in ways that permit evaluation-^and as it 
initiates a review process which measures achievement of those goals and 
objectives, it is automatically in a better position to adapt to external 
changes. 

There may oe other institutional benefits: 

• assurance to student consumers that program qualivj ' .an 

evaluated; 

>> ■ 

• assurance to state government officials, legislators, and the 
• general public that institutions and state higher education 

agencies are accountable; 

• establishment of an ongoing program review and self-evaluation 
process at the institunorl; 

• improvement in institutional and statewide planning; 

• implementation of long-range plans; 

• therapy of self-evaluation. 

'Progra'm reviews may also have an impact on budget requests by identify- 
ing underfunded areas of theiJf=€gram being reviewed. These could include 
areas such as equiprnent, faculty salaries, instructional and support per- 
sonnel, student aid, and general departmental support. 

Educational opportunities available to citizens of a state may be 
enhanced through developiTient of programs in a complementary rather than a 
competitive way when institutions are assigned exclusive roles. In addi- 
tion, more immediate educational responses can be made to state manpower 
needs through cooperative institutional programs. 

Undesirable outcomes of program review nay come about because of 
political pressure.. For example, influential politicians such as state 
legislators, and even hoard members, under pressure from institutions, 
occasionally Subvert decisions to prevent the termination of particular 
programs. As a result of such actions, oth^r underfunded programs suffer 
from limiLed funding. Although institutional staff have sometimes generated 
sufficieht opposition to thwart implementation of program review decisions, 
officials must be convinced that public interest is being served through 
program review processes. 

•Id intenance of the vitality of , pes tsecondary education is the most 
v.i]udt)le outcome to be achieved througn program review, This is possible 
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when programs which are no longer operating effectively are abandoned and 
programs which are vital and productive are strongly supported. 



Following is a section which presents in bn'ef outline form the alterna- 
tive procedures and consequences involved in program review. This section 
is intended as a summary of results of the survey and as a quick guide to 
readers interested in comparing the possible approaches to program review. 



ALTERNATIVES WITHIN PROGRAM REVIEW AND APPROVAL PROCEDURES 

Sources of Authorit y 
-- constitutional 

statutory 
-- policy 

state agjncy agreement with institutions 

Kinds of Authori ty 

recommendatory/advisory 
final decision making 



all postsecondary, undergraduate, graduate and/or two-year 
or less 

majors within programs 
minors within programs 

special izatiop> or emphases within programs 

Lour-ins 

changes in degree requirements for already authorized programs 




programs (curricula leading to a degree or certificate) 



existing and/or new 



vocational and/or academic 



135- 

— aclministrati ve units 

-- research programs 

-- public service programs 

new cosit units appearing in budget 

Purpose of Review 



Existing Proposed 



— to determine harmony wtth institutional 
role and mission , X X 

— to determine harmony with institutional 

and/or state master plan X X 

— to avoid unnecessary duplication X X 

— to make judgments concerning allocation 

of resources X X 

-- to evaluate a program's productivity X 

-- to determine if established standards 

of quality are met ' X 

— to evaluate program's strengths and 
weaknesses X 

-- to provide information necessary to 

improve needed programs X 

-- to determine if resources are adequate 

for quality programs . ' X 

-- to determine if needs justify the programs X X 

-- to determine potential for accreditation X 



Descri ption of Procedures 
Review Process Participants 
Institution 

-- faculty in program 

-- faculty outside program 
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-- students in program 
students outside program 

— alumni from program 

— students who dropped out of program 
-- department/division ghairman 

— dean of college 

"^■^ academic program officer 

institutional research director 
-- trustees 
Sys tem 
-- staff 

* 

interinsti tutional committee 
State agency 

— staff 

\ board members 

Consul tants 

external to institution peers (program related) 
external to institution peers (general) 

Existing Programs 

Criteria for Initial Selection of Programs for Review , 
-- minimum certificates/degf^es awarded* 
minimum enrollments* 

few or no job opportunities for graduates 
questions raised external to the institution 

*$ee state summaries for detailed minimums . 
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-- duplication in state or region 

— general funding limitations I 

— cost of specific program 

— accreditation review schedule 
--established cycle for recurring reviews 

Criteria Used in Evaluating Programs 
Need for program. 

— student interest 

-- demonstration of need generally , 

employment demand 
-- duplication of program in state or region 

— contribution or importance in relation to other campus programs 

— value to society as a whole 
-- inherent value 

- Cost and benefits 

-- number of graduates in past 

projected number of graduates 
-- present enrollment 
-- projected enrollment 
-- cost 

-- student credit hours generated 

-- expected economies from consolidation or elimination 

faculty workload and productivity 
-- faculty quality 
••- funding sources 
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— physical resources? 

-- financial aid availability 
Program objectives ' 

-- harmony with role and mission statement and master plan 
Accreditation 

— potential or achievement of meeting accreditation requirements 

Program Decision Options 

-- unconditional continuation ' 
-- strengthen with additional resources 
-- conditional continuation 

\ 

-- consolidation of one or more programs 
-- termination (immediate or phase out) 
-- suspension 

-- review not completed 

/ 

Locus of Final Decision MakinM 
State agency 
-- staff 

-- governing board or board committee 
Institution 

-- chief executive officer 

academic program officer 

department/division admin.istrator 
-- program review committee: graduate or undargraduate 

governing board 
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Other 

r 

— regional or national group 

— interinstitutional coirmittee 

— state sector staff 
governor or legislature 



Internal (institutional) Procedures for Program Approval and Review ; 
A Suiimary of Survey Results 

Institutional procedures for review of existing and proposed new pro- 
grams are summarized in the following pages. This information was obtained 
by surveying each of the 494 western postsecondary institutions listed in 
the NCES Education Directory, 1977-78 . Institutional respondents were 
asked if they had formal procedures for review of existing and proposed 
acadenic and/or vocational programs and if they had a priority setting 
process for allocation of funds to new or existing programs. In addition, 
they were askeo to send printed materials or a written description of the 
procedures used. Table 1 sumnarizes the initial responses of the insti- 
tutions. 

Table 1 

Summary of Postsecondary Institutional Responses by Level 



Levels of 
Insti turion 


Total 
Surveyed 


Non 
Respondents 


Formal Program Review Procedures 
Yes . No DeveloDinq 


Graduate 


181 


60 


35 


29 


7 


Undergraduate 


91 


50 


24 


16 


1 


Two- Year 


222 


74 


84 


59 


5 


TOTAL 


494 


X 184 


193 


104 


13 



The 193 institutions which /indicated th^it they had formal program review 
procedures were then sent a questionnaire asking for more detailed information 
about their procedures. Responses were verified by telephane interviews with 
designated contact persons at selected institutions. 
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Internal Program Approval 

Program approval is the process^ whereby new program proposals are 
reviewed and decisions are made-' regarding the approval or nonapproval of 
program- proposals. A "typical" process of program approval within col- 
leges and universities is shown in Figure 1. 



Figure 1 
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Typically, the process begins at the departmental level and if approved 
at that level progresses through the institution's faculty governance and 
administrative levels until it is either turned down or approved (sometimes 
the proposals are sent back for further development). 

Our survey results indicate that cai.ipus administrators and faculty 
.are the persons mst frequently involved in the development of internal 
program approval and review procedures.. Other participants involved in 
policy development are shown in Table 2. 
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" Table 2 

Involvement in Development of Current Procedures 
for Review in WICHE States 

Total N 105 ' 
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Number 66 36 9 74 27 32 23 14 26 6 
Rank (2) (3) (9). (1) (5) (4) (7) (8) (6) (10) 



Final disposition of program approval at the institutional level usually 
consists of approval by the campus administration followed by an action of 
the institution's lay board (especially if large sums of mpney are reouired 
to implement the program or if initiating the program will result in an ex- 
panded or different role and mijiSion for the institution). 

The process shown in the diagram in Figure 1 is a generalization based 
on the survey results, but the actual processes utilized may vary from this 
pattern. The process outlined is also more typical of four-year colleges 
and universities than of coimiunity colleges, Bible colleges, and private 
proprietary institutions. In thi.s latter group of institutions, the degree 
of faculty involvement in program d^elopment and approval is typically less 
than in four-year colleges an^ universities. 

Another variation in the approval process occurs in those institutions 
which are either a part of a college or university system or in a state in 
which the state postsecondary education commission approves new programs. 
•In these instances, there may be external review procedures which either 
occur simultaneously or follow institutional approval procedures. For 
example, many state postsecondary agencies require that they be notified 
early in the program development process. 

This approval may be^ a formal request to begin planning or merely a 
notification (oral or written) of the institution's program development 
activities. 

The survey results indicate that most institutions approve programs 
through some kind of formal or informal process and only a very few have 
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no process at all. A followrup survey to the latter group Shows that a 
few handle program approvals' in an ad hoc manner (i.e., only utilized when 
magnitude of the program requires it). 

The purposes for conducting the program approval process in colleges 
and universities are shown in Table 3. 



Table 3 

Purpose for Review— Proposed Programs in WICHE Institutions 

Total N = 105 
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The most frequently cited purposes for conducting internal program 
approval are fairly consistent across most types of institutions and in- 
clude (in rank order): (1) to determine if the documented needs justify 
the program; (2) to determine if the program is consistent with the insti- 
tutional role and mission; and (3) to determine if there are sufficient 
resources. The criteria for review are almost always indicators based on 
these three purposes. 

The most frequently noted criteria for program approval with respect 
to need were (in rank order of frequency): (1) justification of need; 
(2) student interest; (3) job opportunities; (4) duplication of other 
existing programs; (5) value to society; and (6) centrality to other 
programs on campus. The cost and benefit criteria in order of frequency 
of response were: (1) enrollment; (2) physical facilities; (3) sources of 
funding; (4) projected graduates; (5) faculty quality; aid (6) faculty 
productivity. (See Table 4. ) 
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^ Table 4 

Criteria Used in Review of New Program Proposals 

M 105 
A. Need for the Program 
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B. Costs and Benefits of the Pronram 
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C. Objectives of Program 



Consistency with 
Role and Mission 



Appropriateness 
to Revised Role 
and Mission 



77 



56 



D. Accrediting Requirements 
71 



*Single most important criterion 
**Single least important criterion 
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The responses from the WICHE states were generally consistent with the 
responses from the national survey. . 

The "key" individuals in the program approval process based on the 
survey responses are: - (1) faculty associated witK progranr; (2) outside 
peer consultants from the same academic discipline of the program under 
review; (3) the department chairperson; and (4) the academic dean, provost 
and/or vice president for academic affairs. The least frequently noted 
participaVits in the approval process were students not associated with the 
1. program and outside peer consultants with general backgrounds (not from the 
academic discipline under review). 

In terms of the magnitude or importance of participation in the pro- 
gram approval process., the program's faculty (they developed the proposal), 
the faculty on internal review confnittees, and the institution's chief 
academic officer were the most important participants. 

Where outside peer consultants are "llsed, .their judgment has consider- 
able weight in the internal approval process, especially when they are per- 
ceived to be "objective." The survey results also showed that most of the 
institutions using outside peer consultants were at least somewhat satisfied 
with their use in program approval. 

• The cost of the internal approval process is borne by a combination of 
program funds (funds from the department proposing the program) or general 
institutional funds, with a large portion being in the form of contributed 
services of faculty and staff. 

Internal institutional program review of'existing programs is not new. 
Some institutions have been reviewing their programs (or a portion thereof) 
for a number of years. Approximately Vc percent of the WICHE institutions 
responding to the survey had policies and procedures for program review 
before 1965. (See Table 5.) 



• Table 5 

WICHE States Date Present Policies 
and Procedures Were Initiated (%) 

Before 1965 1965-69 1970-75 Since 1975 

12^ 12% ■ 33% 43% 



What is new and noteworthy is: (1),the increased number of institutions 
now conducting internal program reviews; (2) the increased comprehensiveness 
of the reviews; and (3) the increasing impact of program review on institu- 
tional planning and budgeting. 
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Approximately one- third of the colleges and universities surveyed have 
formal systems of internal program review for at least a portion of their 
instructional programs. Among institutional types, the larger four-year 
public colleges and universities indicate the most program review activity. 
Program review in the community colleges is largely restricted to vocational 
programs as a requirement for federal vocational education grants unless the 
community college is a part of a system or district or in a state where the 
state postsecondary agency requires or conducts program reviews. The least 
internal program review activity occurs in public and private four-year 
colleges and developing universities, Bible colleges, and proprietary 
instil^utions . • . , ' 

Most colleges and universities, both nationally and in the WICHE states, 
select programs for review on a rotating cycle. A large number of institu- 
tions, however, use a screening process (sometimes called "program audit") 
to select programs in need of a more in-depth review. Those institutions, 
utilizing this type of screening procedure use program enrollments (usually . 
over a five-year period), the availability of jobs for graduates, and the 
cost of the program as the primary criteria for identifying programs in 
need of a more extensive review. (See Table 6.). 

Table ,6 

Criteria for Selection of Programs for Review 

N = 105 
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In the actual review of existing programs, the most frequently noted 
criteria utilized are shown in Table 7. With respect to the need for the 
proijram, student interest in the program, justification of need, job oppor- 
tunities, and value to society were the most .frequently noted (in rank order), 
In the area of cost and benefits of a program, enrollment, cost, faculty pro- 
ductivity, graduates, and faculty quality were frequently cited criteria. 
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Criteria for Review of Existing Programs 

N = 105 
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C_ • Cr.Ltp_i"J ^Jof, Review of Ex istin g Procjram 



Objectives of the Program 
Consisten'.y with Role and 
Mission. Appropriate to 
Revis. d Role and Mission. 
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Most colleges and universities have responded affirmatively to the 
question of v^ether there is a relationship between program review and 
institutional planning and budgeting, Less frequently noted were rela- 
tionships between institutional reviews and regional- accreditation^ pro- 
fessional accreditation, state agency program review and state master^ 
planning. (See Table 8..) / 



Table 8 

Sunmary of the Relation of Program Review \ 
to Other Activities \ 

Total N • 193 





Activity 


Level of 
Insti tutlon 


State 
Master Plannl'g 


State Agency 
Program Review 


Institutional 
Planning 


Institutional 
Budgeting 


Regional 
Accreditation 


Professional 
Accredl tatlon 


Yes 


No 


Yes 


No 


Yes 


No 


Yes No 


Yes 


No 


Yes 


No 


2-yedr 
Public 
Private 


26 
1 


8 
1 


34 
1 


5 
1 


37 

2 


1 


36 2 
2 


32 

2 


2 


28 


4 


4-year 
Public 
Private 


3 


2 
8 


3 


2 
7 


4 

)0 




4 

9 


3 
7 


1 
3 


3 
7 


1 
1 


University 
Public 
Pri Vdte 


22 
1 


1 

14 


20 
2 


4 
12 


20 
17 




22 
15 


16 
16 


7 


19 
13 


6- 
4 


Total 
Public 
Pri vate 


51 
2 


11 
23 


57 
3 


11 
20 


61 
29 


1 


62 2 
26 


51 
25 


10 
3 


50 
22 


11 
5 


Total 


33 


34 


60 


31 


90 


1 


88 2 


76 


13 


72 


16 



Final decisions (if any) on institutional program reviews generally 
rest with the chief campus academic officer o>" somewhat less frequently 
with the institutional program review committee. 

State- level Review Procedures 



State agency procedures- for review of existing and proposed new programs 
are sunmarized in t.ie following pages. This information was obtained from 
materials provided by the state agencies and through personal interviews on- 
site and at regional conferences with agency officials. The state procedure 
outlne has been reviewed and approved by the agency administrator respon- 
sible for program review in each of the respective wes ^rn states. 
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ALASKA 



Authorlt A^ 

Alaska Postsecondary Commission 
Statutory 



Review of existing junior and senior Irvel proarams 
Approval of proposed programs at junior and 'sen. or level 



Action 



Recofmendations reaaraing new program approval -to institutions and to Governor 
and leqislature 

Recommendations about continuarice, discontinuance, consolidation of existing 
prqqrams. 

Scope of Program Review 

At present, no review of programs which are^ not offered by a traditional institution of higher 
education, but plans for this type of review contemplated. 

All 4-year and 2-year institutional prog/am offerings in both , the public and private sectors 
of both graduate and undergraduate levels. 

Only degree-credi f programs reviewed. 

To advise and assist University of Alaska and private institutions In identifying programs 
Conmission sees as questionable, with recomnendations for action. 

To inform Governor and legislature of recommendations when no action taken or disagreement exists. 

To encourage consolidation of weak programs, termination of Inappropriate programs , and creation 
of needed new programs 

Descrrpt jon of Procedures t 

Use of consultants: 

Consultants not presently used. 

Criteria for selection of programs to be reviewed: 

Low graduate productivity 
Low and/or declininq enrollment 
Conparati vely (within'the state) hiah cost 
Low manpower nepds 

Duplication at bf^flr more campuses within the state 

Specialized programs not consistent wi th institutional roles, location, or capabilities 
J lime-lme for review: 

Not settled. Three-year review anticipated, possibly settling to a five-year when 
system matures further. 

Relationship with external reviews:' 

Presently, all accreditation teams from the Northwest Accreditina Association which visit 
Alaska institutions include a member of the Postsecondary Commission as advisory member. 

Fundinq for conducting program review: 

Ongoing primary responsibility of Commission supported mainly by state general fund money, 
but also to some ef'xtent by Title XII money. 

Ddt'a compilation for pronram review: ^ j 

Each school provides Commission with the following data either through documents already 
submitted or by special request: 

A list by name, degree, school, and major program of all graduates each yi^r 
An updated list of all offerings whether new, combined, or deleted 
Degree proyram enrollments 
Budget d/Ua to each department and/or program 
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Manpower projections are also obtained from the Alaska Department of Labor and the 
Institute for Social, Economic, and Governmental Research, University of Alaska. 

Criteria used in evaluating programs: 

A- Quality of faculty 
' Facilities allocated for use 
Library support 
-- Faculty workloads 
Faculty-student ratio 

Issues or Problems Encountered 

— Data not completely available from all schools 

— Defining quality ^ 

/lot enough time, money, or staff to review highly questionable programs or-site 

— Disagreements between Institutional personnel and system personnel as to what the "official 

university data really is 

— Lack of a central system review on existing programs 

— Lack of university guidelines, regarding presentation of new program requests 

\ 
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ARIZONA 



Authority 

Board of Regents * ♦ 

Consti tutional 

T- Review of existing programs at colleges and universities 
Approval of proposed programs at colleges and universities 

Action 

— Final decision 

Arizona Board of Education and State Board for Community Colleges of Arizona ^ 
Policy 

— Review of vocational and community- college programs 

Action 

Final decision 

Scope of Review 

Public 4-year college and university programs 

All programs consisting of a series of courses and activities leading to a degree - 
or certificate 

— Larger structural units such as College of Education 

Coirmunity college and vocational programs 

Review of all vocational programs annually 

Purpose 

Review is conducted to determine: 

— Duplication of programs 

Quality (which may lead to identification of steps needed to bring a program up .to 

a minimal gual ity) 
Cost-effectiveness 

Action needed to establish new programs, strengthen existing programs, continue programs 

at status guo, modify existing programs, or discontinue existing programs 
Ways to improve both managenent and programs at ir^titutional level 

Description of Procedures ' 

Approval ^,of all new degree programs 

Proposals developed by Institutional faculty and administrators 
Submitted to Board of Regents In regulred format which includes: 

Description of program 
Projected enrollment 
Faculty needs 
Library needs 

— Facilities and eguipment needs 
Administration needs 
Eligibility for accreditation 
Support orogram needs 

Costs 

Evaluation plans 

Community college 

New curriculum and vocational course approval must be obtained from the State Board 

for Community Colleges 
All vocational programs undergo annual assessments by a visiting assessment team 
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Existing programs 

7- State Board of Regents staff and Academic Officers Committee review data 
f submitted by institutions 

Criteria for selection of programs for systemwide review: 

— Number of graduates from the program In each of the past five years as 

compared to national averages for comparable programs 

— Number of students enrolled in the program in each of the past five years 

— Size of classes and cost of courses including types of costs and sources 

of funding 

Cost per program Including marginal costs 

— Program quality as reflected by its regional or national reputation, 

faculty qualifications, and the level of the first position achieved ' 
by graduates of the program 
Faculty workloid and productivity 

— Total nunter of graduates in the program from all institutions in the 

state, region, and nation 
Economies and improvements in quality, if any, to be achieved by elimination 

and/or consolidation of the program 
Demand trends for the program and any other evidences of general student 

interest 

Appropriateness of the program to institutional role or mission 

— Library and other support facilities available to the program 
Service and teaching load of faculty participating in the program 
Importance of program to other components of the university 
Local, regional, and national needs for graduates of the program 
Age and stage of development of the program 

(In addition, all programs are to be reviewed internally on a ten-year cycle by 
each institution,) 

' Institutions conduct self-study of programs selected for review 

Statewide review by external consultants who receive final appointment from 
Board of Renents from lists prepared jointly by institutions and state agency 
staff 

Over several weeks, the consultants visit each campus which offers the 
program under review, to meet with faculty, students, cormiunity members, 
and representatives of professional organizations 

Criteria used for evaluation of program quality 

Resources available to program 

Outcomes in terms of need for, placement of, and adequacy preparation 

of graduates in program 
Program costs 

Other relevant factors, e.g., unusual state needs 

— Institutions review consultants* reports before submission to Board of Regents 
Consultant reports reviewed and decision made by Academic Affairs Conmittee and 

Long Range Planning Cofiinlssion of the Board of Regents 
Review procedure conducted in about, three months with three cycles per year * 
Funds appropriated to the Board of Regents by the legislature cover review 

expenses for consultants 

Possible decisions 

Strengthen 

— Modify 
Continue intact 
Phase out/discontinue 
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CALIFORNIA 

Authority 

California Postsecondary Education Commission (CPEC) 
Statutory 

Review of proposed academic and occupational programs ^ 
Action — reconrendation \ 
— Review of existing programs \ 
Action recommendation / 

Preparation and annual update of five-year master plan for California 
postsecondary education 

Scope of Review ( 

— All vocational programs 

— All academic programs defined as a series of cou!^ses arranged in- a scope and sequence leading 

to a degree or certificate 

Purpose*, and descriptions of program review procedures will be discussed separately for each of the three 
segments and th.e Commission. 



California Postsecondary Education Commission 



Purpose 



To insure articulation among segments * 
To prevent unnecessary duplication of programs 

To coordinate and evaluate program review efforts among educational segments ' 
To determine respX)nsi veness to student and societal heeds 

To suggest procedures/guidelines ifor segmental reviews and to ascertain that such reviews 
are occurring 

Des cription of the Procedure 

Segments submit program review reports to CPEC 

University of California System 

Purpose • V , 

To preserve diversity within the system ^ ' 
— To promote creative autonomy in planning lacademic programs 

To insure that all academic activities ar& subject to similar standards of objective scrutiny 
To maintain qual i ty 

Dc scriptio n of the Procedure 

Approval of proposed new graduate programs 

••- Proposed programs must be reviewed and approved by department faculty, dean. 
College Graduate Study Committee, Academic Vice President, and the Steering 
Committee of the Academic Planning and Program Review Board before being 
submitted to the System office 

Review of existing programs 



Standing sub-committees of the Graduate Council 
Ad hoc faculty committees 
External reviewers 
-- Students 
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Criteria for selection of programs to be reviewed 

— Cyclical method for periodic review of all programs 
Identification of "problem" units 

Evaluation criteria 

— Research stature of faculty 
Quality of students 
Placement of graduates 
Details of program 

California State University and Colleges (CSUC) 



To insure effective and efficient utilization of available resources 
To maintain quality of existing programs 



Description of the procedure 

Approval of proposed new programs 

Proposal developed at campus level and submitted to the Division of Academic Program 
• and Resource Planning in academic year preceding projected implementation 

Evaluation criteria 

--(^Financial support 

Qualified faculty 
- Physical facilities 

Library holdings 

Review of existing programs 

Department and school conduct self-study 

Program reviewed by University Graduate Council or Planning and Educational 

Policies Council 
Distinguished colleagues from off-campus review program 

Attempts made to coordinate reviews with professional and regional accreditation 
visits 

Criteria for selection of programs to be reviewed 

One-fifth of all programs reviewed each year 
Evaluation criteria 

Goals and emphases 

Admission practices and student characteristics 
Advisement procedures 
Operation of the program 
Quality and relevance 

California Community Colleges 



Purpose 



To monitor educational programs and selected services in California community colleges 
To determine the conformity of a program with district policy and plans and with the statewide 
master plan 

Description o f the procedure 

Approval of proposed new programs 

Extensive planning and study at the college and district levels precede submission 

to Chancellor's office 
Consistency with college academic master "plan which is reviewed by conmunity advisory 

groups, faculty, curriculum committee, and administration, is determined 
Approval by District Board of Triistees is required 
— Chancellor s office approval required 

CPEC decides if it wtl'l approve Chancellor's recpmnendatlon 
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^valuation criteria 

Scope of instruction in accordance with legislation 

— Appropriate review has taken place \ 
Four-year institution has agreed to accept transfe^^of students 

— Need for occupational programs justified by job marR^t 
Student interest in enrolling \ 
Realistic plan for providing necessary resources for ^i^ality programs 

Review of existing programs 

— Self-study by campus administrators, faculty, students, counselors, advisory 

conmittee members \ 

— Staff in Chancellor's office \ 
Review team from other cofimunlty colleges \ 

\ 

Criteria for selection of programs to be reviewed \^ 

— All vocational programs annually \ 
Evaluation criteria 

— Same as criteria used for program approval 
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COLORADO 



Authority 

Colorado Cormission for Higher Fducatlon (CCHE) 
Statutory 

Approval of new programs 
Action final decision 

Review of existing programs 
Action — recommendation 



In 1978 General Assembly directed CCHE to review programs for preparation of 
nursing personnel and education personnel and to develop a state Master Plan 
for postsecondary education. 

Scope of Review 

5^^^.!^^ 0^ nev programs Including "any new curriculum which would lead to a new degree program 
or the esuabl ishment of a college, school, division, Institute or department." 



Purpose 



To promote realistic academic planning at Institutions 
— To provide input to the determination of statewide priorities 
To provide a framework for statewide academic program planning 



Description of Procedures 



New programs 



Criteria 



Proposal developed and approved at institution 

Submitted to CCHE where 1t goes to other institutions 1n the state with similar 
programs for review and cofinent and, if requested by any institution, to the 
Advisory Council on Higher Education Programs 

Relation to the long-range education plan for the state 
Relation to the over-all academic mission of the Institution involved' 
Impacts the program on other institutions. Including private ones 
Institutional capability to offer the program 

Availability of the program to students who desire work in the field 

Desirability of attractino students to a particular institution 

Interest of the administration and staff in the establishment of such a program 

CCHE also consider^^ statewide goals 

To maximize post-high school education op]rortun1 ties for qualified youth of the state 

— To promote diversity in statewide higher education 
To encourage. high quality in all programs 

- To achieve full coordination cff education program efforts among the institution^^ 
To insure an orderly pjttern o\ enrollment growth in public higher education 

To effect the best uti 1 ization \of available resources 

T i me - 1 me 

New program approval or disapproval votediby CCHE in January, May, and September 
usually within 90 days after proposal received.. 

Note: Bill of Particulars provided for all disapproved programs 
and will hold a hearing '^if requested. 

Existing programs 

Colorado Commission for Higher Education 

Procedure currently being developed 

Approved programs not offered for a period of two years and approved new 
new programs not Implemented within one year are removed froin future 
institutional and Comnlssion publications and must go through the usual 
approval process before being offered 
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Outcomes 



State Colleges dnd University Consortium procedures for program review 
Goals 

To improve the effectiveness of academic pru rams through 
self.study 

To assure that academic programs are responsive to the broad 

educational needs of Colorado citizens 
To avoid needless duplication of facilities and programs In 

higher education Institutions in Colorado 
, To provide a sound base for long-range educational planning 
To assure that essential educational services are being 

effectively provided through the best utilization of available 

resources 

Use of external consultants optional 
-7 Self-study by Institution uses broad-based institutional committee evaluation 

submitted to President and Vice President for Academic Affairs 
Systemwide task force conducts systemwide reviews and comparisons to evaluate 

effectiveness of comparable programs of Consortium Institutions 
Task force recommendations to Consortium cabinet 

State Board for Community College, and Occupational Education 

Approves or disapproves all vocational program proposals 

Nursing — specific recoimendations by CCHE to Increase access, to balance supply to manpower 
needs, to seek and maintain accrf.-ditation, and to efficiently use state resources 

Teacher Eaucdtion specifir recormendations by CCHE to improve these programs in Colorado 
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HAWAII. 

AuthorltY ^ 

Board of Regents (constitutional) 

Statutory ^ 

— Review of all existing programs 
Approval of all new programs 

Action — final decision 

Scope of Program Review 

- University of Hawaii at Manoa Academic Senate In the early 1970's established policy and 

procedures requiring review of each academic program 
Review and evaluation of each established Instructional, research, and public service program 
every fifth year - Board of Regents policy (1973). New programs authorized as provisional 
to be evaluated when the first students graduate - no tenure decisions or cormltments during 
the provisional status, * 

- Definition of program (not clearly stated or applied) some campuses - organizational units; 

others ^ degree majors. Process does not look at overall state picture, only individual 
programs . 



Purpose 



Mixture of purposes and objectives. 



— CManoa) Maintain and protect program quality for programs that were to be kept and 

discontinue programs which should not be kept - primarily .liagnostic link with 
WASC self-study for accreditation carried out with program review, 

- (Board of regehtsJJJjMtlfy programs that are weak or high cost and low priority 

which shoiad7|Wp|g^ 

Description of Procedures - ' ■'^iiv:- 

Campus - academic senateV pr^ram faculty /students, and department chairman; Graduate Council, 

If applicable; Associate. D^an and Dean; Chancellor 
State-Level - system staff, Board of Regents 

Consultants - Visiting accreditation team used as consultants at Manoa. Consultants make 
recommendations based on quantitative review. 

Criteria 

« 

Program objectives 
. Priorities 

.Target groups ' 
Costs 
Funding 
Facilities 
Faculty 

Measures of effectiveness 
Continuing need/demand 

Responsibility for developing the schedule and for submitting an updated version annually to 
the Office of the President delegated to each campus 

Time- 1 1 ne 

Updated version of review submitted to Board of Regents and Of/ice of the President annually 
— Process of reviewing one program allotted one year to complete 

Relationship wi th External Review 

Manoa linked with WASC through concurrent accreditation and self-study review of programs 
Annual visiting comnlttees from WASC focus on programs reviewed for year of the visit 



Funding 



Very costly in tenir, of psychological and physical energy due to uncertainty, and in fiscal 
costs, coordinating duties, report processing, and undue program attention and focus (versus 
campus-wide focus) by visitation comnittees 
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Data Compilation 

— Absence of operational criteria to be applied in program review 
Decisions 

— Continue/discontinue programs on program^by'^program basis 

Problens 

Extended and complicated process making reviews as much as ftve years old when President's 
office receives them ^ 

— Mixture of purposes and objectives reduced value to institution 

— Behavioral response not calculated carefully enough for steps to Increase probability of 

positive implementation - faculty feel it is simply another ritual imposed on them to 
serve bureaucracy 

' — Lack of plan and priorities results In piecemeal approach without looking at the entire campus 

Absence of operational criteria at policy level 

Absence of review policies for administrative and support functions so review of programs 

implies threat of discontinuance 
Program review of one-fifth of programs each year obscures priorities and reallocations 
Program review of one- fifth of programs leaves constant state of uncertainty 
Accreditation processes and annual review link have reduced effectiveness of both 
Classes of action resulting from program review should be clearly known by all fwrticipants 
Program review based on unit and sub-unit evaluations gives no opportunity to look at curricular 
needs and programs across the system. Two kinds really needed: 

1, DepartJnental internal program review \ 

2. Quantitative review by outside consultants \^ ' 

University now involved in multicentered review and analysis of past five years to modify 
current approach 
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IDAHO 

Auth.tr J tj' 

'^tJle Bo.ird of Education 
Statutory 

- Approvd.l of proposed four-year college and universi'ty programs 
Action final decision 
Review of existing four-year college and univers\ty programs 
Action final decision 



Pol U.y 



Approval of proposed community college programs 

Action recommendations 
Review/ of existing community college programs 
• Action recommendations 



''Zall7Ztr^^°^''''' ^'"^'"^ ^° ' '^'^'^'''^'^ ^^JO'-. field of specialization 

-- New schools, colleges, .nstitutes, or foundations 

, sting program areas which result in a degree or certificate awarded by the institution 
A)! graduate programs 



" ^^r-;" quality of higher education through curriculum coordination, planning, and 

" T't i.r J* f'H Procedure 

Af. 'jf ripw proqrans 

-- New program proposal developed by faculty and department administrators within 
institution and submitted to irs ti tutiona 1 curriculum comnittee 

-- Noticp of intent sent at same time to State Curriculum Cormittee (Institutional 

Vki' Presidents) through Deputy Director of Curriculum Planning 
- ^urritulum Comnittee members report reactions, from their respective campuses to 
•he Depaty Director 

-- -rstitutional reoresentati ve explains proposed program to Curriculum Conmittee 
-- .<''r:.t ce approved proposals added to Deputy Director's cumulative list submitted 

^ to Board of Education at June meeting with his recormendations for action 
-• ."-.titutions submit funding requests for approved programs at July Board meeting 

- Jroqrams for which Board grants permission to seek funding are assigned priority 

ir, the higher education budget requests to the next legislature 

- jeered program proposals may be rewritten and resubmitted to Curriculum Cormittee 
After two rejections, State Board approval may be sought independent of Curriculum 
LOfrmittee approval 

Vocational program proposals follow same procedure with the State Director for 
Vocational Education presenting the proposal 

Lo^iunity colleges Voluntarily submit proposals for new programs to the Curriculum 
lumrii t tee 

- ^'rivdte institutions invited to pa'-ticipdte in approval procedure via their 

academu: vilP presidents 

-<'»'<'w -tf jli LMisting po . tsecondary programs 

<>lMity (Jirector examinee, annual productivity d<Ua and reports programs not meeting 
'innimum standr^rds to Boc^rd of Education 
. s^'k'cts tMrpp or four program areas for review during the r^-.t y^^r 

- ^^.^f 1 t /t)..f dl staff condu ts self>study 
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Consultants selected jointly by Deputy Director and institutional staff spend 

full week visiting institutions to meet with self-study author, faculty, students, 

and administrators; holding hearings (.t which all interested persons make presentations 

Consultant reports incorporated by Deputy Director into official report for State Board 
rev lew 

Criteria for selection of graduate prograios for review (no review of any graduate programs triggered 
to date) 

Minimum number of degrees awarded 

Master's degree programs 5 graduates per year average for 3 years 
Doctor ' s, degree programs 3 graduates per year average for 5 years 
Professional certificate and specialist programs 

5 graduates per year average for 3 years 

Programs which have had complaints or problems 

Board iden^ificatiun of 3-4 program (Srfeas for review each year 

Evaluation criteria 

Objectives of the program--recent changes, anticipated changes 

Accompl ishments 

Plans for the next five years 

Job placement fo previous three years 

Job market supply and demand 

Relationship with institutional role and mission 
Similar programs in thv.- state and in the region 

Interest by outs^'de groups \ 
Er»rollment stat^ stics--pas t and projected, capacity 
Degrees grant^^d in prior ten years 
Personnel --facul ty and st?ff 

competenci es-- existing , larkirci 
Curriculum 

present courses, courses to be added ,. streng' n, unique features 

def icii-ncies, remedies, interrelation with other curricuja 

anticipated changes 
' ■ Informa t ional reso jrces/1 i brary/fac i 1 i ties/ equipment 

present, needed 
Finance • 

total cost, source 
-- Accreditation 
-- Continuinq education 
- - f ' r 1 0 1 1 1 f? s 



' stdt^'wide proqram reviews 

Department of Architecture 
Parks and Recreation 

Bu^.iness 

Continuing Lducation 
Jour na 1 1 sni 



1 ■• Ngrs 1 n(\ 
\ hj' l\ t. I Of 



1 



aboli shed 



new focus of journalism 
program offered by Boise 
State - a cOTrmuni cations 
>j major 

one state publication for 
i] 1 three universities 

reconnended changes at asso- 
ciate level at Lewis-Clark 
State College and delayed 
implementation of' baccalaureate 
degree p^^ogram 

authorized Energy Experiment 
Station at Idaho State University 
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MONTANA 



Author 1 t y 

Board of Regents f'jr Higher Education 
Constitutional and statutory 



Approval of new programs 
-^Review of existing programs 



Action 



Final decision 



Scope of Review 



All proposed new schools or colleges 

All series of courses arranged in ? scope or sequence leading to a certificate or a degree 
which has not been offered in the Institution or appeat'^d in the catalog in the prpvious 
two years 



Purpose 



Bettor utilization of resources 

Improvement of quality /' ^ 

Descrip tion. of Procedures / 

New programs (includes vocational programs) 

Proposals developed and approved at institution 

Proposals submitted to the Commissioner of Higher Education 

Comnissioner^s staff reviews proposal and makes recommendations to the 

Curriculum Coimilttee of the Board of Regents 
Board may require use of external consultants 

Cn tei^ia 

— Quality 
Objectives 

' Resw#rces available (faculty, facilities, equipment, library holdings) 
Costs K 
^ Projected enrollments 

Job opportunities for graduates 
Duplication 

Existing programs \ 

Each insU^tion is to review its programs on a regular basis 

Regional and professional, accredi tation provides review of Some programs 

Selected lateral (inter-institutional) reviews conducted 



Outcomes 



fkj reviews conducted since 1975-76, pending completion of Role and Scope statement for the 

University System , 
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NEVADA 



Authority 

University of Nevada System Board of Regents (constitutional) 
Statutory 

Approval of new program proposals for vocational -technical , and public 
senior and junior institutions 
Action final decision 
Review of existing 'programs 
/ Action final decision 



S cope of Review 



Purpose 



Instructional, public service, and research programs at public junior and senior Institutions 
Series of courses which lead to a decree or certificate not previously awarded 
New major or emphasis with an existing program 

New departments or divisions, schools or colleges, laboratories, centers, or similar administrative 
units 



To work toward the improvement of the quality of the services required to meet the goals 
identified in the four-year plan 



Description of the Procedu re 

Approval of new programs 

Basic information developed by institution and submitted to Board of Regents 
Board decides if additional work should be done 
-- Detailed information presented and Board rejects or approves proposed program 
subject to 'l egislati ve funding 
Institution has option to select and use consultants at institutional expense 



Criterfd for evaluation 

Program description and objectives 

Relationship of objectives to other segments of the institution 

Need for program, local, state, regional, and national intrinsic academic value, 

employment opportunities 
Resources 

Entrance and graduation requirements 
Course content--changes needed 
Accref^itation requirements 

Outcomes 

4S proposed programs were conside'^ed by the Board of Regents between 1969-1977; 16 have been 
funded and implemented 



Prob 1 ems 

Consultants to Institutio s may be advocates for program rather than unbiased evaluators 
Inadequ'itp definition may invite political intervention 

Greater stdtr agency Involvement would permit programmatic review across institutions 
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NEW MEXICO 



Aut hority 

Boarc: of Educational Finance (BEF) 
Statutory 

, Approval of new program proposals 

Review of existing programs 

Action 

Final decision 

Scope of Review 

All postsecondary programs defined as a .eries of courses leading to a degree which Identifies 
graduates (does not include options within programs) at all vocational-technical, public junior, 
and senior institutions 

All new, different, renamed, or reorganized graduate majors that 'require additional staff or 

other resources submitted to the BEF. Staff determine if BEF approval is needed, 
BEF maintains an "Inventory of Majors" 

Purpose 

— To insure more thoughtful planning and more, rational setting of institutional priorities 
To enhance interinstitutional cooperation 

Qpspri ption of the Procedur e ^ 

Approval of new graduate programs (Ed,D. and Ph.D. programs must be received two years before 
implementation; master's programs, one year before implementation) 

Proposals developed and approved at institution 

Submitted in early concept form to BEF for approval to proceed with development and 
again as fully developed proposal 

Criteria for evaluation ^ 

Institution provides: 

Description of t^e proposed program 
Relation to existing programs 
Additional courses required 
Evidence of need 
-- Types of employment for which graduates are qualified 

— Costs 

State Board provides: 

f — Report of similar programs offered In the state - ^ . 

— Number o( degrees granted in the program in New Mexico for past 8 years 
New Mexico's relation to the national picture in production of degrees 
Student load in comparison to that at other state institutions 

Credit hours per instructor 
Average class size by level 

■» 

Existinq programs 

Procedure followed 1975-77: 

Procedure designed by BEF and statewide Academic Council for Higher Education 
comprised of, the chief academic officers of the six public institutions of 
higher learning 

Fact sheet for every program developed by staff at e^ch Institution 

Academic Council developed criteria for selection of programs that should 'undergo 

extensive review 
BFF staff developed an inventory of degree programs 

Academic Council visited each institution and met with faculty and staff concerning 

programs which did not meet the. criteria for selection 
Institutions produce accurate data on program enrol Irnent, graduates, and costs on 

ongoing basis 
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Criteria for selection of programs to be reviewed 

(Criteria are reviewed and established by staff and council for each cycle) 
Complete review of all programs every five years 
Graduates 

Number of graduates by degree level from each program In the 
last five years 

Baccalaureate level programs having four or less graduates per 

year on the average over the last five years 
Master's level programs having three or less graduates per year 
. on the average over the last five years 
Specialist level programs having three or less graduates per 

per year an tfie average over the last five years- 
Doctoral level programs having two or less graduates per year 
, on 't^^e average over the last five years 

Enrol Iment 

Average of head count students enrolled In Fal/fl974 and Fall 1975' 
Baccalaureate level programs having twelve or Te^s majors per 

year on the average over the last two reporting periods 
Master's level programs having nine or less majors' per year on 

the average over the last two reporting periods 
••- Specialist level programs having nine or less majors per year 

on the average over the last two reporting periods 
Doctora\ level programs having six or less majors per year on 

the average over the last two reporting periods 

Time-ime 



Complete process takes two years 

Source of funding 

- No special funds appropriated; part of regular operation of institution 
Criteria for evaluation 



Out' of'Ps 



Developed by. the Council for each cycle 



141 programs failed to meet criteria' 
Intensive review resulted in: 71 dropped or combined 

21 continued 

21 required continuing review 



\ 

) 
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OREGON 

Authority 

Oregon Educational Coordinating -Commission (OECC) 
Statutory 

Revieurf of new vocational program proposals 
Review of all proposed new postsecondary programs 
Review of existing programs 

Action final decisions limited to cases referred by state boards 

State Board of Higher Education (SBHE) 

Statutory 

Approval of new program proposals 
Review of existing programs 

Action final decision except where there is adverse intersegmental impact 
State Board of Education Conmunity College Division * 
Statutory 

Approval of proposed new vocational programs 
Action final decision except where there is adverse intersegmental impact 

Scope of Review 

New majors, degrees, and certificate programs 
New areas of specialization or options for existing programs 
Changes in degree requirements for already authorized programs 
- Program defined as "collection of activities and resources contributing to the 
^^^ucation of a group of students pursuing a common curricular path" 



^ ' Oregon Educational Coordinating Comiiiission 

Purpos e 

To determine if the proposed program or change will have an adverse intersegmental impact 
To determine consistency with statewide goals and objectives 

* 

P-^jAQP,^ Jo^ 0^ th e Procedure 

Review of proposed new programs 

V Program proposals approved by the Department of Education, Cormunity College 

Division, and proposals approved by th^ State Board of Higher Education 
submitted with supporting documentation for review by the Coordinating Cormission 

Evaluation criteria ' 

Consistency with statewide plan 
Interseot^^ntal coordination 

Review of existing programs 

Segmental boards submit program review repurts to OECC 



Purj)o_se 



S U te Bo^d_o_f_Hl9.h pr Edu c ation 



To avoid unnecessary and unwise duplication of programs in Oregon colleges and universiM'es 
(program approval ) 
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To assure that programs are consonant with .he needs of the state «nd consistent with the 
goals and objectives of the Institution (program approval) 
--To determine the strengths and weaknesses as well as the benefits and effectiveness of 
institutional programs (program review) 
To stimulate and encourage Improvement in quality of programs (prograir. review) 

Description of the Procedure 

Approval of new program proposals 

Proposal developed and approved by institution involving faculty, students, 
administrators, advisory boards, interested groups and the public 

Proposal^, and supporting documentation forwarded to the office of the State Board 
of Higher Education where the CQncern is segmental planning and segmental manage- 
ment of governance. 

Board^s office analysis and recommendation are forwarded with the proposal to the 
State Board of Higher Education for consideration and action 

If the proposal is approved by the State Board of Higher Education, copies of 
the material considered and the action are fdrwarded to the Oregon Educational 
Commission where statewide planning and intensegm^ntal coordination is emphasized 

Board or Conmission may involve consultants in procedure 

Evaluation criteria 

Community and societal need (including employment needs, where relevant) 
Resources to offer program 

Relationship to other programs in the institution 

Duplication and impact on other institutions and segments 

Fiscal impact and priorities * 

Review of existing programs 

Conducted and feicilitated by the staff of the Board's Office of Academic Affairs 
Recommendations prepared by the Office of Academic Affairs and the Chancellor for 

Board consideration and action 
Information about programs under review provttfeS'by appropriate departments and 

and university officials 
Students' perceptions solicited through questionnaire surveys 
Outside consultants and evaluators used to assess the quality of programs under 

review and the general "climate" for learning and research within the department 

Criteria fo^ selection of programs to be reviewed 
Low degree conferral 

Programs qffered by more than one fnstitution 

f 

Evaluation criteria 

Description of program did statement of program objectives 
Enrollments and student credit hour production 
Student profile 

Resources utilized in offering program ^ 
Faculty profile 

Time required for completion of the degree 
Student attitudes toward program 
Program outcomes such as job placement of graduates 
Cost-benefits interaction 
• Quality 

Relationship to other institutional programs, and to the community and larger 
area to be served 

- Prospects for the future of the program 

Effects of discontinuing the. program ^ 

Possit 1e decisions 
Continue 

Continue on a conditional basis " 
Consolidate with one or more other prqflrams 

— Suspend ir terminate 
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Outcomes 



1 976--Reviewed 37 low-degree-conf erral graduate programs 
14 continued 

3 continued on a conditional basis " 
5 consolidated 

4 Suspended or terminated 
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Purpose 



Physics, chemistry, biology master's and doctoral level; continuation for all 

programs authorized with areas in need of improvement identified 
Business administration; continuation for all programs 

19«8-79 (in process) 

Political science, sociology, economics, geography and urban studies 
Education 

1979-80 (plans) 

Matnematics, geology, computer science, statistics, systems science 

1977-79--'Development of comprehensive report on Graduate Education in Oreqon--A Response to 
a 19^7 Legislative Budget Note ; better understanding on the part of legislators and 
th^ publ ig of the nature and importance of graduate education in Oregon 

State Board of Education Community College Divisvon 

To determine the need for a program ^^^s-^ 
« To analyze such factors as duplication, program impact, fiscal impact, program congruence 
wi th state goal s 
To determine adverse impact on other segments of education 

P-gJJ'iJ^^P/', Q^. th e Procedure 

Approval of proposed new programs 

Proposals developed and approved at institution involving faculty, administrators, 

advisory committees, and interested groups 
— Proposals and supporting documentation forwarded to segmental representative 

for use. in informing all other community colleges 
If application data appears relatively complete, a copy is sent to the staff 

of OECC for informal communication about proposed programs 
Manpower analyst in Conwunity College Division reviews and analyzes manpower data 
Division program specialist analyzes application in detail and recommends approval 

or disapproval 
State Board acts on proposal 

Approved proposals with supporting documentation submitted to OECC 
Division approval of curriculum provides authority to implement the program 

Lv^luation criteria 

Fi nancia 1 impact 
^^"^"^nsistency with established guidelines 
--■ Description and objectives 
O(kupdtional demand 
Intersegmental impact 
jp/1cdtion of orograms 

Pevipw 0^ existing programs * 

Local coriTiU'iity colleges have adopted procedures and criteria 
Federal regulations require evaluation of existing vocational programs every 
f \ ve years 
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Evaluation criteria 

-« (Final criteria currently being developed by Department of Education) 
f-inancial 

Agre^iment with established guidelines 

Assurance that program remains current 

Consistency of hours and courses with approval documents 

Identification of situations which require special contracts or arrangements 

Lack of sex discrimination ^ 

Reflection of federal requirements 

Consistency with statewide objectives and institutional role 
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UTAH 



Authority 



Board of Regents 

Statutory 



Approval of new programs 
Review of existing programs 



Action final decision 



Scope of Review 



Purpose 



Whenever a new cost unit is to be added, the Board reviews it 

All postsecondary programs--series of courses which lead to a degree, diploma, or certificate 
New schools, colleges, institutes, foundations, and administrative units 



To eliminate unnecessary spending 'caused by program duplication 
To assure that available resources are adequate to produce quality education 
— To isolate programs not meeting selected minimum criteria and to determine if the services 
they provide are sufficient to merit attention 



Description^ of the Procedure 

Approval of new programs 



Development of concept for proposed program submitted to Board of Regents' office 

Review of insti*:utional proposal by other institutions in Utah System of Higher Education 

Board staff assesses proposal In relation to institution's role and in terms of statewide 

planning and invites Institution to submit complete proposal depending on the assessmenll 
Panel of outside consul tar^ts, thoroughly reviews all new advanced professional and doctoral 

degree program proposals 

) , I 



Need (manpower requirements^ projected enrollments) 
State's ability to finance 
-- Costs 

Proposed curriculum 
Needed facilities and equipment 
' Consistency of program w th the institution's role 

I 

Tim^r-linp ^ ' i 

Request to approval is about three months 
Exi' ting programs 

Board of Regents staff use established criteria to identify-p/ograms for review 
— Self-study of identified programs conducted by institutional staff 
Board of Regents make final decision about program status 

Criteria for selection of programs to be reviewed 

Examine programs in areas identified by Regents, or in Master Planning efforts 

ExMiine all degree programs every third year for productivity according to HEGIS 
retort information 

Ideinify for Indepth review programs that do not produce a three-year average, by 
dpgree level as follows: 

Number of Graduates Per Year 
Edu cational Degree Level (Three-Year Average) 



1 . Doctoral Programs 

2. Master's Programs 

3. Bachelor's Programs 



4 . Assoc late Programs 

5. Certi f 1catf*s 



5 (if doctorate offered) 
3 (if no doctorate offered) 
8 (if- master's/doctorate offered) 
5 (if no graduate work offered) 



7 
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Criteria for evaluation of programs 



Numbers of program graduates and enrollees In each of the last 3 years 
Program class sizes for the preceding year 
Program cost per SCH during the past 3 years 
Faculty workload data 
Evidence of program nullity 

jtate and national production data in the program discipline 

Dollars saved and quality improvements to be achieved through program consolidation 

or el imination 
Student interest, manpower demand, and placement 
Increased dollars necessary to establish a viable p»-ogram 
Other selected infamation 




Possible Regents' decisions 



Continuation without Prejudice 
Probation 

Termination of programs 



Outcomes 



10 



DEGREES, PROGRAMS, AND ADMINISTRATIVE 
UNITS DISCONTINUED 



Insti tut ion 



Total 
1959-1975 



Total 
1975-1979 



Total 
1969-1979 



University of Utah 

Utah State University 

Weber State College 

Southern Utah State College 

Snow College 

Dixie Colfege 

College of Eastern Utah 

Utah Technical College-Salt Lake 

Utah Technical Col lege-Provo 



71 
37 
21 
6 
3 
6 



2 
7 



10 
4 



2 
5 

10 
15 
2 



01 
41 
22 
8 
8 
16 
15 
9 
3 



Totals 



153 



203 



Count on Programs Denied/Disapproved 
1969-1979 



Total number proposed programs disapproved by the Utah State Board 
of Regents during the period 1969-70 through 1978-79 equals 71. 
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WASHINGTON 



Author i ty 



Council for Postsecondary Education (CPE) 
Statutory 

Review and recofnnendatlon of new programs 
Review of existing programs 



Action reconmendations to institutional governing boards, governor, and legislature 



State Board for Corrinun^ College Education 
Statutory 



Approval of new vocational programs 
" Review of existing programs ' 



Action final decision 



Scope of Review 



Purpose 



"''^^one'gTE^JJcJSISS)^'^' '° ' °^ "^^^'^-^^^ (State Board for Co^uhity 

-- Review and reconriendation of new program proposals 

Proa^m°[c'r'^!"^/''°^''"^ concentrated at the oraduate level 

"rrLrc^ns^^^SleS^tlretro^^ ^ ^ and .eludes new 

'MlTuon '''^ coordination among the parts of a complex system of J^'stsecondary 



— To faciVitate planning for postsecondary education in the state 
Description of the Procedure 

Approval of new program proposals 

^ I - Proposal developed at the institution submitted to CPE preceded by a preliminary 
planning statement at least six months earlier ^^eu*?a oy preliminary 

Council staff review proposal and verify data ' 
Institutional staff react to Council recomnendatians 

Criteria for evaluation of proposal 

Description of program 

Objectives s 
Content 

Evidence of program need 

Student: 

Student inquiries and applications 
— Alternative programs at the institution 
Career orientation 

In^ti tutional : / 

Interrelated programs at the institution 
Faculty development 

Institutional priority of program ared 



Soc letd 1 



Carepr market andlysis 
Social problem solving potential 
Similar programs in the state and region 
Y External agency support (st.Up, regional, and national) 
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Program qual i ty 



Curriculum (national standards) 

faculty (existina and new) 

^creditation (program and related programs) 

Piacefnent of graduates from related programs 



Institutional fit 



Institutional role and mission 
-•- Fit with other institutional programs and functions 

Fiscal impact 

Staffing (faculty, administrators^ etc.) 
Facilities 

Support (libraries, Computer, etc.) 

Decisions must be made at all points (both internal and external) on the relative 
merits of a proposal and the balance of quality, need, cost, and institutional fit. 
The relative weighting of the elements will depend upon the nature of the proposed 
program. 

Existing programs 

Postsecondary Education Council staff 
Institutional administrators and faculty 
Institutional governing boards 

Criteria for selection of graduate programs to be reviewed 

Chronic low productivity 
Duplicated programs 

Criteria for evaluation of graduate programs 

Program need (student, institutional/faculty, societal) 
Program costs, resources, productivity 
Harmony with institutional role 



Poi.sible decisions 




Outcomes 



rogram- continuation 
Contingent continuation 
Conditional continuation 
Consol idation 
Temination 
Revie'w not complete 
— Other 



500 programs reviewed 
83 programs phased out 
118 programs to receive further review 
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WYOMINti 

' % ' ') 1 leqe i on^,} ion 

-- Approval of nev/ pr'onrdni'. 
f^teview of eAi^.ting pfoyrams 

' - - final de.. 1 s > on 



•1 ' procj'-j'v. 

^ which It^, 1 to d certificate or degr-pf' 
'f rt.-ucnrrj internal a(\'rova] ot all new program proposals 



an..! yrj'luate procjrams developed and approv. ^ at University of Wyoming 
r ,r,jj.,fe L'.nrittee. Graduate Advisory Cormittee, Academic Planning Comnittee, 
> tn.H of Acader.iL Aftairi,, and Board of Trustees 
y^^j.- colle'i^' and voLdti'^nal programs developed ana approved by campus staff 
,M,-,f rolle.iM V] vocarionai program proposals submitted to Cormiunity College 



' . t 1 VP . 



>if 'T'^ ; r , ')f,|" in, , J, I'll r; ) 



••'^i'" j-v^'-.ity profi^d-)'. 1'. sf'lf-st;jdy for e>ternal accreditation 

■ J. I'^fin-ii prr .'jr jnr, wn ]ud»'S rPVH'w Of : 



Imp* ji»ny f tJuc(jttgn In Tlie Wpst 



WICHE. the Western h>terstate Commission for Hi^'h v 
er Education, Js a rudHc regional organization. It helps- 
the thirteen member states cooperatively provide 
high*quallty;*cost'eftective-programs to meet lhe;edu- 
^cation and manpower needs of the West. AVI^HE*$. 
Project on Expanding Regionsil Cooperation in Gradu- 
ate and Professional Educatibri encourages resource 
sharing in graduate and professional educatioh in the 
West by providing information about these programs 
throughout the region. It is supported by a two-year 
gfUfU trorn the Carnegie Corporation of hew York and 
by WICHE stcjte dues through its Student Exchange 
Program. / 



|p NCHEMS 



/ 



MCHEMS. the National Center for Higher Education 
Management Systems, is an independent, nonprofit 
organization working nationally to improve planning 
and management in postsecondary education. The 
NCHEMS four-year project on state-level information 
was undertaken In 1975 with funding from the W.K. 
Keilocjg Foundation and supplemental support from 
the National Center for Education Statistics to assist 
state-level planners for postsecondary education in 
addressing their information needs. Project results 
have been gathered 'in documents providing a refer- 
enre framework and planning guidance to help state 
agencies develop an (nformation-system planning 
approach tailored to individual tatc responsibilities 
and needs. 



